CN112416950A - Design method and device of three-dimensional sketch structure - Google Patents
Design method and device of three-dimensional sketch structure Download PDFInfo
- Publication number
- CN112416950A CN112416950A CN202110093365.3A CN202110093365A CN112416950A CN 112416950 A CN112416950 A CN 112416950A CN 202110093365 A CN202110093365 A CN 202110093365A CN 112416950 A CN112416950 A CN 112416950A
- Authority
- CN
- China
- Prior art keywords
- weight
- bucket
- edge
- node
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000013461 design Methods 0.000 title abstract description 12
- 230000006870 function Effects 0.000 claims abstract description 31
- 241000406668 Loxodonta cyclotis Species 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 14
- 230000035772 mutation Effects 0.000 claims description 8
- 238000010276 construction Methods 0.000 claims description 4
- 238000010845 search algorithm Methods 0.000 claims description 4
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 4
- 230000010365 information processing Effects 0.000 claims description 2
- 230000002829 reductive effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 238000003491 array Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000002441 reversible effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2264—Multidimensional index structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a design method and device of a three-dimensional sketch structure, computer equipment and a storage medium. The method comprises the following steps: the method comprises the steps of constructing a three-dimensional sketch structure for storing stream data information, dividing each bucket in a three-dimensional bucket array into three areas, obtaining the stream data information of a graph structure, and updating the stream data information into the three-dimensional sketch structure according to hash functions corresponding to different depths. The invention combines the representative key reservation and the majority voting, the constructed three-dimensional sketch structure simultaneously maintains the structure information and the weight information of the graph data, one-time updating is carried out on the primary stream data, the constant probability error range is ensured, the majority voting algorithm is used for selecting the stored keys so as to record the most representative graph edges, the error range can be reduced, and the reversibility is realized so as to improve the query efficiency.
Description
Technical Field
The application relates to the technical field of computer networks, in particular to a design method and device of a three-dimensional sketch structure, computer equipment and a storage medium.
Background
In the past decade, graph structures have been used extensively to model complex structured data in interactive applications, such as network traffic and social networks. As a sequence of data that changes over time, the stream data that preserves the graph structure may continuously describe entities (e.g., social media users) and connections between entities (e.g., interactions between users) and form the basis for various services such as network anonymity detection, community discovery, and the like. However, analyzing massive amounts of streaming data in real-time is quite challenging. For example, each link in a large ISP or data center processes about millions of data packets per second. In the face of such a situation, the conventional data structure (e.g., adjacency table) is not suitable for storing the graph structure in stream data.
In the relevant scenario of streaming data applications, rather than sacrificing efficiency to arrive at an exact result, it is more desirable to get an approximate result quickly. To meet such a requirement, sketch is widely used, and generates a summary in a space-saving manner, performs approximate weight estimation, and can query elephant streams (heavy-hitter) or mine top-k terms (top-k items). A Sketch is a stream data aggregation structure that can store a fixed number of entries in units of buckets, and classical sketches (e.g., Count Sketch, K-ary Sketch, and Count-Min Sketch) linearly project stream data into a lower dimensional space that preserves data aggregation features using multiple hash functions.
Some previous graph sketch techniques (e.g., GSS, TCM, gMatrix) improved the structure of the classic sketch, recording stream data that preserves graph structure in a generic way, supporting real-time updates and queries. The gSketch is designed for edge frequencies, but it can only answer queries based on edge frequencies, and not more complex graph structure queries. TCM and GSS aim to preserve graph structure and support various types of queries, but they cannot do reverse hash queries by traversing the entire key space to obtain the edges or nodes of interest, or storing other index tables to record keys and their hash values. gMatrix recovers a key by pruning the key space using modular hashing techniques, but the process of recovering a key using modular hashing techniques requires lifting entries in the sub-key space, an enumeration that is computationally expensive.
Therefore, the prior art cannot take both the estimation accuracy and the query efficiency into consideration.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method, an apparatus, a computer device, and a storage medium for designing a three-dimensional sketch structure, which can achieve both estimation accuracy and query efficiency.
A method of designing a three-dimensional sketch structure, the method comprising:
constructing a three-dimensional sketch structure for storing stream data information, wherein the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
dividing each bucket in the three-dimensional bucket array into three regions, respectively storing the sum of weight values mapped to all edges of the bucket, a key with the largest current weight mapped to the bucket and a value of an indication counter for indicating to reserve or replace the key with the largest current weight, and initializing the bucket;
acquiring stream data information of a graph structure, and calculating a bucket index of each edge at each depth according to node information of the edge in the stream data information and hash functions corresponding to different depths;
updating the bucket of each depth corresponding to the edge according to the bucket index, wherein the updating comprises: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and carrying out statistics and query on the stream data according to data stored in the three-dimensional sketch structure.
In one embodiment, the method further comprises the following steps: according to the edge in the stream data informationThe bucket index ofUpdating the bucket of each depth corresponding to the edge, wherein,the depth sequence number of the packet,is composed ofThe hash function corresponding to the depth layer comprises the following steps:
updating the sum of the weight values in the bucket to be:
wherein,is the weight information of the edge in question,is an edgeMapped toThe sum of the weight values in bucket of the depth layer;
wherein,is an edgeMapped toThe key with the largest current weight in the bucket of the depth layer;is an edgeMapped toA value of the indication counter in a bucket of a depth layer;
judgment ofPositive and negative of (2)When the temperature of the water is higher than the set temperature,the value of (d) remains unchanged; when in useAt the same time, updateAndcomprises the following steps:
in one embodiment, the method further comprises the following steps: and carrying out statistics and query of flow data according to data stored in the three-dimensional sketch structure, wherein the query comprises side weight query, node weight query, elephant flow side query, elephant flow node query, mutation flow query, mutation node query, subgraph weight query and path reachability query.
In one embodiment, the method further comprises the following steps: counting and inquiring flow data according to the data stored in the three-dimensional sketch structure, wherein the step of inquiring the edge weight is as follows:
Is compared to a depth ofkIn a bucket ofValue of (A) andwhether they are the same or not, if so, the edgeIn thatkThe edge weight estimate for the depth layer is:
according to the edgeThe minimum value of the edge weight estimated values at all depths is obtained to obtain the edgeThe edge weight estimate of (a) is:
In one embodiment, the method further comprises the following steps: and carrying out statistics and query on flow data according to the data stored in the three-dimensional sketch structure, wherein the node weight query step is as follows:
obtaining a node to queryx;
For depthMiddle row ofIf each packet ofThen nodexIn thatkDepth layerjThe output weight estimates for the columns are:
otherwise, the nodexIn thatkDepth layerjThe output weight estimates for the columns are:
wherein,representing the total depth of the three-dimensional bucket valuesDegree;representing the total column number of the three-dimensional bucket numerical value;is a nodexIn thatkDepth layerjAn output weight estimate for the column;
according to the nodexIn thatkDepth layerThe output weight estimated value of the column is obtained to obtain a nodexIn thatkThe output weight estimate for the depth layer is:
according to the nodexIn thatThe estimated value of the output weight of the depth is obtained to obtain a nodexThe node weight estimate of (a) is:
for depthInIf each packet ofThen nodexIn thatkDepth layeriThe input weight estimates for the rows are:
otherwise, the nodexIn thatkDepth layerjThe input weight estimates for the columns are:
according to the nodexIn thatkDepth layerInput weight estimation value of the column to obtain a nodexIn thatkThe input weight estimate for the depth layer is:
wherein,the total row number of the three-dimensional bucket numerical value is represented;is a nodexIn thatkAn input weight estimate for the depth layer;
according to the nodexIn thatThe depth input weight estimated value is obtained to obtain a nodexThe node weight estimate of (a) is:
according to the nodexOutputting weight estimatesAnd input weight estimatesTo obtain a nodexThe node weight estimate of (1).
In one embodiment, the method further comprises the following steps: performing statistics and query of stream data according to data stored in the three-dimensional sketch structure, wherein the sub-graph weight query step is as follows:
obtaining a subset of nodes of a subgraph to be queriedWhereinnRepresenting a number of nodes in the subset of nodes;indicating a serial number ofiThe node(s) of (a) is (are),;
determining a subset of edges of the subgraph from nodes in the subset of nodesWherein m represents the number of edges in the subset of edges,;indicating a serial number ofiThe edge of (a) is provided with,;
Will be provided withThe result of the sum of the weights of all edges inAs weight estimates for the subgraph.
In one embodiment, the method further comprises the following steps: and carrying out statistics and query on stream data according to the data stored in the three-dimensional sketch structure, wherein the step of path reachability query is as follows:
according to each bucket in the three-dimensional sketch structureSum of the weight values in (1)Summing to obtain the total weight of the data stream in the current periodF(ii) a WhereinSequence numbers representing the bucket in three dimensions of row, column and depth;
traverse allWhen is coming into contact withAt the same time, readKey stored inObtained by edge weight queryThe edge weight of whenWhen it is determinedIs an elephant stream edge; whereinIs a preset threshold parameter;
at all weights greater thanThe elephant stream edge is subjected to path reachability query through a breadth-first search algorithm, wherein the path reachability query is to determine whether the source node a can pass through the weight of at leastIs connected to the target node b.
An apparatus for designing a three-dimensional sketch structure, the apparatus comprising:
the device comprises a three-dimensional sketch structure construction module, a data information storage module and a data information processing module, wherein the three-dimensional sketch structure construction module is used for constructing a three-dimensional sketch structure for storing stream data information, the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
the Bucket initialization module is used for dividing each Bucket in the three-dimensional Bucket array into three regions, respectively storing the sum of weight values mapped to all edges of the Bucket, a key with the largest current weight mapped to the Bucket and a value of an indication counter used for indicating to reserve or replace the key with the largest current weight, and initializing the Bucket;
the bucket index determining module is used for acquiring stream data information of a graph structure, and calculating a bucket index of each edge at each depth according to node information of the edge in the stream data information and hash functions corresponding to different depths;
a bucket updating module, configured to update the bucket of each depth corresponding to the edge according to the bucket index, where the updating includes: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
and the counting and inquiring module is used for continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and counting and inquiring the stream data according to the data stored in the three-dimensional sketch structure.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
constructing a three-dimensional sketch structure for storing stream data information, wherein the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
dividing each bucket in the three-dimensional bucket array into three regions, respectively storing the sum of weight values mapped to all edges of the bucket, a key with the largest current weight mapped to the bucket and a value of an indication counter for indicating to reserve or replace the key with the largest current weight, and initializing the bucket;
acquiring stream data information of a graph structure, and calculating a bucket index of each edge at each depth according to node information of the edge in the stream data information and hash functions corresponding to different depths;
updating the bucket of each depth corresponding to the edge according to the bucket index, wherein the updating comprises: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and carrying out statistics and query on the stream data according to data stored in the three-dimensional sketch structure.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
constructing a three-dimensional sketch structure for storing stream data information, wherein the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
dividing each bucket in the three-dimensional bucket array into three regions, respectively storing the sum of weight values mapped to all edges of the bucket, a key with the largest current weight mapped to the bucket and a value of an indication counter for indicating to reserve or replace the key with the largest current weight, and initializing the bucket;
acquiring stream data information of a graph structure, and calculating a bucket index of each edge at each depth according to node information of the edge in the stream data information and hash functions corresponding to different depths;
updating the bucket of each depth corresponding to the edge according to the bucket index, wherein the updating comprises: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and carrying out statistics and query on the stream data according to data stored in the three-dimensional sketch structure.
According to the design method, the device, the computer equipment and the storage medium of the three-dimensional sketch structure, the three-dimensional sketch structure used for storing the stream data information is constructed, each bucket in the three-dimensional bucket array is divided into three areas, and the sum of weight values of all edges mapped to the buckets, the key with the maximum current weight mapped to the bucket and the value of an indication counter used for indicating to reserve or replace the key with the maximum current weight are stored respectively; acquiring stream data information of a graph structure, updating the sum of weighted values mapped in a bucket according to side information in the stream data information and hash functions corresponding to different depths, updating the value of an indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the side information, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter. The three-dimensional sketch structure constructed by the invention simultaneously maintains the structure information and the weight information of the graph data, updates the primary stream data only once, has a constant probability error range, uses a majority voting algorithm to select a stored key so as to record the most representative graph edge, can reduce the error range and realize reversibility so as to improve the query efficiency.
Drawings
FIG. 1 is an application scenario diagram of a design method of a three-dimensional sketch structure in one embodiment;
FIG. 2 is a schematic diagram of a three-dimensional sketch structure in one embodiment;
FIG. 3 is a flow diagram illustrating the updating of values in a packet according to one embodiment;
FIG. 4 is a block diagram of a device for designing a three-dimensional sketch structure in one embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The design method of the three-dimensional sketch structure provided by the application can be applied to the following application environments. Constructing a three-dimensional sketch structure for storing stream data information, dividing each bucket in a three-dimensional bucket array into three regions, respectively storing the sum of weight values mapped to all edges of the bucket, a key with the maximum current weight mapped to the bucket and a value of an indication counter for indicating to reserve or replace the key with the maximum current weight, and initializing the bucket; acquiring stream data information of a graph structure, updating the sum of weighted values mapped in a bucket according to side information in the stream data information and hash functions corresponding to different depths, updating the value of an indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the side information, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter.
In one embodiment, as shown in fig. 1, there is provided a method for designing a three-dimensional sketch structure, including the steps of:
102, constructing a three-dimensional sketch structure for storing stream data information, wherein the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions.
The sketch data structure is a probability statistical data structure with high reliability, wherein the characteristics (occurrence frequency and data stream size) of a large number of elements in stream data need to be counted, a hash function is used for multiple times for the stream data to map an event to the frequency, the size of the event is estimated within reasonable deviation, and the memory occupation is obviously reduced. Sketch may model the stream data as a (key, value) form, where a "key" is a field in one or more data packet headers, such as a source address or a combination of a source address and a destination address, and a "value" is a stored characteristic, such as the number of data packets.
The graph structure referred to in the stream data is a discrete structure formed by nodes and edges connecting the nodes, and the graph structure may be represented by G = (V, E), where V represents a node set, E represents an edge set, and a relationship between any two nodes in the graph structure is represented by an adjacency matrix, when no weight is referred to in the graph structure, an element of the adjacency matrix is 0 or 1, 0 represents that the corresponding node has no relationship, and 1 represents that the corresponding node has relationship; when weights are involved in the graph structure, the element values of the adjacency matrix are weight values between nodes.
Classical sketch is designed as a two-dimensional bucket array and cannot query for structure-based information (e.g., path and subgraph based queries), so applying classical sketch methods directly to stream data would lose graph structure information.
The three-dimensional sketch structure DMatrix provided by the invention expands the sketch structure into three dimensions, and the DMatrix is a structure with three dimensionsh×h×wThe first two dimensions of the three-dimensional sketch structure of each bucket represent the length and the width of the bucket array, and the third dimension represents the depth of the bucket array. Use of DMatrixwIndependent hash functions, each defining a hash function from a node in the set of nodes to 1,h]a mapping of integers within a range. The two-dimensional arrays with the same depth are set to have the same length and width, and the same hash function is used for the same depth.
And step 104, dividing each bucket in the three-dimensional bucket array into three regions, respectively storing the sum of the weight values of all edges mapped to the buckets, the key with the maximum current weight mapped to the bucket and the value of an indication counter for indicating to reserve or replace the key with the maximum current weight, and initializing the buckets.
In order to realize reversible query, the method of the invention reserves a space in the bucket for storing keys of edges or nodes. Due to the hash collision problem, different edges may map into the same bucket. Therefore, there may be multiple keys in the same bucket. For this problem, the LD-Sketch selection uses the associative key array in the extended bucket to accommodate more candidate keys, but the cost of dynamic memory allocation is very high. To save computation and storage overhead, DMatrix retains only one key in each bucket. For buckets with hash collisions, we will select a key to store in the bucket. In fact, the bucket stores the set of weights for all edges mapped to this bucket that have the same hash value, and the key selected by the present invention to store should be most representative of this aggregated result. Therefore, the present invention stores the most weighted key in this bucket. In selecting the stored key, we apply a majority voting algorithm (mjty), which records the most weighted key in the bucket. The mjty processes a sequence of votes and attempts to find a majority of the votes. In the single vote voting process, it will store the candidate majority votes observed so far in the stream and an indication counter that records whether the currently stored votes are still candidate majority votes.
We useThe representation is located atiGo to the firstjColumn depth ofkIn which,. As shown in fig. 2, the memory area of such a packet is divided into three parts: the first block isRecording the sum of all edge weight values hashed to the bucket; the second block isStoring the key with the largest current weight of the bucket; the third block isIt is an indication counter that checks whether it should be reserved or replaced as in mjtyA stored key.
And 106, acquiring stream data information of the graph structure, and calculating the bucket index of each edge at each depth according to the node information of the edge in the stream data information and the hash functions corresponding to different depths.
Obtaining stream data information of graph structure according to edges in the stream data informationNode information ofxAndyand hash functions corresponding to different depths, calculating the bucket index of the edge at each depth asWhereinthe depth serial number of the bucket is;is composed ofAnd the depth layer corresponds to a hash function.
The update procedure for DMatrix is as follows: for incoming weight offIs not limited byFirst, calculate the index of bucketThen will beValue increase of regionfAnd checkWhether the key stored in (1) isIf so, it will indicate a counterIncrease in value offOtherwise, it willValue of (2) is reducedf. If it is notTo a value of less than 0, we useIs replaced byAnd will store the key inIs set to its absolute value.
And step 110, continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and performing statistics and query on the stream data according to data stored in the three-dimensional sketch structure.
Operations for querying on stream data information of the retention graph structure can be classified into 4 types: edge-based queries, node-based queries, subgraph-based queries, path-based queries. The edge-based query is divided into: side weight query, elephant stream (heavy-viewer) side query, mutation stream (heavy-changer) side query; node-based queries are further divided into: node weight query, elephant flow (heavy-hit) node query, mutation flow (heavy-change) node query; the subgraph-based query is mainly a subgraph weight query, and queries the total traffic weight in a subgraph composed of all nodes and connecting edges thereof in a node set V; the path-based query is mainly a path reachability query, and queries whether a communication path exists between the node a and the node b.
In the design method of the three-dimensional sketch structure, the three-dimensional sketch structure used for storing stream data information is constructed, each bucket in the three-dimensional bucket array is divided into three regions, and the sum of weight values mapped to all edges of the bucket, the key with the maximum current weight mapped to the bucket and the value of an indication counter used for indicating to reserve or replace the key with the maximum current weight are respectively stored; acquiring stream data information of a graph structure, updating the sum of weighted values mapped in a bucket according to side information in the stream data information and hash functions corresponding to different depths, updating the value of an indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the side information, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter. The three-dimensional sketch structure constructed by the invention simultaneously maintains the structure information and the weight information of the graph data, updates the primary stream data only once, has a constant probability error range, uses a majority voting algorithm to select a stored key so as to record the most representative graph edge, can reduce the error range and realize reversibility so as to improve the query efficiency.
In one embodiment, the method further comprises the following steps: as shown in fig. 3, according to the edge in the stream data informationBucket index ofUpdating the bucket of each depth corresponding to the edge, wherein,the depth number of the packet is,is composed ofThe hash function corresponding to the depth layer comprises the following steps: updating the sum of the weight values in the bucket according to the weight information of the edge as follows:(ii) a Wherein,is the weight information of the edge(s),is an edgeMapped toDepth of fieldSum of weight values in bucket of a layer; comparing the key with the largest current weight in the bucketAnd edge: when in useWhen, the value of the update indication counter is:(ii) a Wherein,is an edgeMapped toThe key with the largest current weight in the bucket of the depth layer;is an edgeMapped toA value indicating a counter in a bucket of the depth layer; when in useWhen, the value of the update indication counter is:(ii) a Judgment ofPositive and negative of (2)When the temperature of the water is higher than the set temperature,the value of (d) remains unchanged; when in useAt the same time, updateAndcomprises the following steps:
in one embodiment, the method further comprises the following steps: and carrying out statistics and query on flow data according to data stored in the three-dimensional sketch structure, wherein the query comprises edge weight query, node weight query, elephant flow edge query, elephant flow node query, mutation flow query, mutation node query, subgraph weight query and path reachability query.
In one embodiment, the method further comprises the following steps: counting and inquiring flow data according to data stored in the three-dimensional sketch structure, wherein the step of inquiring the side weight is as follows: obtaining edges to query(ii) a Is compared to a depth ofkIn a bucket ofValue of (A) andwhether they are the same or not, if so, the edgeIn thatkThe edge weight estimate for the depth layer is:(ii) a Otherwise, the edgeIn thatkThe edge weight estimate for the depth layer is:(ii) a Wherein,is an edgeIn thatkAn edge weight estimate for the depth layer; according to the edgeThe minimum value of the edge weight estimated values at all depths is obtained to obtain the edgeThe edge weight estimate of (a) is:(ii) a Wherein,for queried edgesThe edge weight estimate of (2).
In one embodiment, the method further comprises the following steps: counting and inquiring flow data according to data stored in the three-dimensional sketch structure, wherein the step of inquiring node weightComprises the following steps: obtaining a node to queryx(ii) a For depthMiddle row ofIf each packet ofThen nodexIn thatkDepth layerjThe output weight estimates for the columns are:(ii) a Otherwise, the nodexIn thatkDepth layerjThe output weight estimates for the columns are:(ii) a Wherein,representing the total depth of the three-dimensional bucket value;representing the total column number of the three-dimensional bucket numerical value;is a nodexIn thatkDepth layerjAn output weight estimate for the column; according to the nodexIn thatkDepth layerThe output weight estimated value of the column is obtained to obtain a nodexIn thatkThe output weight estimate for the depth layer is:(ii) a Wherein,is a nodexIn thatkAn output weight estimate for the depth layer; according to the nodexIn thatThe estimated value of the output weight of the depth is obtained to obtain a nodexThe node weight estimate of (a) is:(ii) a Wherein,is a nodexThe output weight estimate of (a);
for depthInIf each packet ofThen nodexIn thatkDepth layeriThe input weight estimates for the rows are:(ii) a Otherwise, the nodexIn thatkDepth layerjThe input weight estimates for the columns are:(ii) a Wherein,is a nodexIn thatkDepth layeriAn input weight estimate for the row; according to the nodexIn thatkDepth layerInput weight estimation value of the column to obtain a nodexIn thatkThe input weight estimate for the depth layer is:(ii) a Wherein,the total row number of the three-dimensional bucket numerical value is represented;is a nodexIn thatkAn input weight estimate for the depth layer; according to the nodexIn thatThe depth input weight estimated value is obtained to obtain a nodexThe node weight estimate of (a) is:(ii) a Wherein,is a nodexThe input weight estimate of (1); according to the nodexOutputting weight estimatesAnd input weight estimatesTo obtain a nodexThe node weight estimate of (1).
In the directed graph, nodes have an in degree and an out degree, the sum of edge weights of an in node is referred to as an in weight, and the sum of edge weights of an out node is referred to as an out weight. The inflow weight and the outflow weight of the nodes can be respectively calculated through the three-dimensional sketch structure provided by the invention.
In one embodiment, the method further comprises the following steps: counting and inquiring flow data according to data stored in the three-dimensional sketch structure, wherein the step of inquiring the subgraph weight is as follows: obtaining a subset of nodes of a subgraph to be queriedWhereinnRepresenting the number of nodes in the node subset;indicating a serial number ofiThe node(s) of (a) is (are),(ii) a Determining a subset of edges of a subgraph from nodes in the subset of nodesWhere m represents the number of edges in the subset of edges,;indicating a serial number ofiThe edge of (a) is provided with,(ii) a Obtaining weights of all edges in a subset of edges by an edge weight query(ii) a Will be provided withThe result of the sum of the weights of all edges inAs weight estimates for the subgraph.
In one embodiment, the method further comprises the following steps: counting and inquiring flow data according to data stored in the three-dimensional sketch structure, wherein the path reachability inquiry comprises the following steps: according to each bucket in the three-dimensional sketch structureSum of weighted values inSumming to obtain the total weight of the data stream in the current periodF(ii) a WhereinSequence numbers representing the three dimensions of row, column and depth of the bucket; traverse allWhen is coming into contact withAt the same time, readKey stored inObtained by edge weight queryThe edge weight of whenWhen it is determinedIs an elephant stream edge; whereinIs a preset threshold parameter; at all weights greater thanThe elephant stream edge carries out path reachability query through a breadth-first search algorithm, wherein the path reachability query is to determine whether the source node a can pass through the weight of at leastIs connected to the target node b.
In one embodiment, the method further comprises the following steps:counting and inquiring stream data according to data stored in the three-dimensional sketch structure, wherein the step of elephant stream side inquiry comprises the following steps: given threshold parameterTo query all elephant streams, first the total weight of all streams for that epoch is calculatedF. Then, we examine each bucketIf, ifLet us orderAnd query using edge weight query operationsReturn all of satisfactionIs not limited by。
In one embodiment, the method further comprises the following steps: counting and inquiring flow data according to data stored in the three-dimensional sketch structure, wherein the step of inquiring elephant flow nodes is as follows:
the elephant flow on the node outflow weight is taken as an example. Given threshold parameterFirst, the total weight of all streams in the period is calculatedF. Then, the sum of each row in the bucket array for different depths is checked, for the secondiGo if, ifThen give an orderAnd query the operation using the node weightReturn all of satisfactionNode (a) ofx。
In one embodiment, the method further comprises the following steps: counting and inquiring stream data according to data stored in the three-dimensional sketch structure, wherein the step of abrupt stream edge inquiry comprises the following steps: given threshold parameterTo query all burst edges, first obtain all different edges from the bucket arrays of two adjacent epochsAnd calculating the total flow change of two adjacent periods,Is to the true total flow variationIs estimated, andsince hash collisions will counteract weight changes in different directions. Then, the user can use the device to perform the operation,andrespectively representUpper and lower limits of (d). Using the results returned from the edge weight query to assign an upper edge weight limit, i.e.. For a depth ofkSet of bucket,Edge ofMapping to bucketIf, ifThen give an orderOtherwise, to. The edge weight lower bound is then assigned using the maximum of the edge weight lower bounds for the different depth estimates, i.e. the edge weight lower bound is assigned a value. Order toAndrepresents the last epoch edge weightUpper and lower limits of (1), ofAndrepresenting current epoch edge weightUpper and lower limits of (d). Estimated edgesIs given by:. Finally, each edge appearing in two adjacent epochs is examined. For each edge, ifOrThen calculate an estimate of the change in weight thereofIf, ifReturning edgeConsider an edgeA burst stream appears above.
In one embodiment, the method further comprises the following steps: carrying out statistics and query on stream data according to data stored in the three-dimensional sketch structure, wherein the step of querying the mutant stream nodes is as follows: given threshold parameterTo inquire about allThe method comprises the steps that firstly, all different nodes are obtained from bucket arrays of two adjacent periodsAnd calculating the total flow change D ̃ of two adjacent periods.
Then, orderAndrespectively representUpper and lower limits of (d). Next, the node outflow weightFor example, a mutant stream node query operation is illustrated. Using the result returned from the node weight query to assign an upper limit to the node weight, i.e.. For a depth ofkSet of bucketNode ofxMapping to bucket arrayIn, ifThen give an orderOtherwise, to. Then, order. Assigning the lower node weight limits using the maximum of the lower node weight limits for different depth estimates, i.e.. Order toAndrepresents the node weight of the last periodUpper and lower limits of (1), ofAndrepresenting node weights for the current epochUpper and lower limits of (d). The estimated maximum weight change for node x is given by:. Finally, each node present in two adjacent epochs is examined. For each node, ifOrThen calculate an estimate of the change in weight thereofIf, ifReturning nodexConsider a nodexA burst stream appears above.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 4, there is provided a design apparatus of a three-dimensional sketch structure, including: the method comprises a three-dimensional sketch structure building module 402, a bucket initializing module 404, a bucket index determining module 406, a bucket updating module 408 and a statistics and query module 410, wherein:
the three-dimensional sketch structure constructing module 402 is used for constructing a three-dimensional sketch structure for storing stream data information, wherein the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
a bucket initialization module 404, configured to divide each bucket in the three-dimensional bucket array into three regions, respectively store the sum of weight values mapped to all edges of the bucket, the key with the largest current weight mapped to the bucket, and a value of an indication counter used for indicating to reserve or replace the key with the largest current weight, and initialize the bucket;
a bucket index determining module 406, configured to obtain stream data information of the graph structure, and calculate a bucket index of each depth of the edge according to node information of the edge in the stream data information and hash functions corresponding to different depths;
the packet updating module 408 is configured to update the packet of each depth corresponding to the edge according to the packet index, where the updating includes: updating the sum of the weight values in the bucket according to the weight information of the edges, updating the value of an indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edges, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
and the statistics and query module 410 is configured to continuously obtain stream data information, update information of all sides in the stream data information to the three-dimensional sketch structure, and perform statistics and query on the stream data according to data stored in the three-dimensional sketch structure.
The packet update module 408 is further configured to update the edge according to the stream data informationBucket index ofUpdating the bucket of each depth corresponding to the edge, wherein,the depth number of the packet is,is composed ofThe hash function corresponding to the depth layer comprises the following steps: updating the sum of the weight values in the bucket according to the weight information of the edge as follows:(ii) a Wherein,is the weight information of the edge(s),is an edgeMapped toSum of weight values in bucket of depth layer; comparing the key with the largest current weight in the bucketAnd edge: when in useWhen, the value of the update indication counter is:(ii) a Wherein,is an edgeMapped toThe key with the largest current weight in the bucket of the depth layer;is an edgeMapped toA value indicating a counter in a bucket of the depth layer; when in useWhen, the value of the update indication counter is:(ii) a Judgment ofPositive and negative of (2)When the temperature of the water is higher than the set temperature,the value of (d) remains unchanged; when in useAt the same time, updateAndcomprises the following steps:
the bucket update module 408 is further configured to perform statistics and query on stream data according to data stored in the three-dimensional sketch structure, where the step of side weight query is: obtaining edges to query(ii) a Is compared to a depth ofkIn a bucket ofValue of (A) andwhether they are the same or not, if so, the edgeIn thatkThe edge weight estimate for the depth layer is:(ii) a Otherwise, the edgeIn thatkThe edge weight estimate for the depth layer is:(ii) a Wherein,is an edgeIn thatkAn edge weight estimate for the depth layer; according to the edgeThe minimum value of the edge weight estimated values at all depths is obtained to obtain the edgeThe edge weight estimate of (a) is:(ii) a Wherein,for queried edgesThe edge weight estimate of (2).
The bucket updating module 408 is further configured to perform statistics and query on stream data according to data stored in the three-dimensional sketch structure, where the node weight query includes: obtaining a node to queryx(ii) a For depthMiddle row ofIf each packet ofThen nodexIn thatkDepth layerjThe output weight estimates for the columns are:(ii) a Otherwise, the nodexIn thatkDepth layerjThe output weight estimates for the columns are:(ii) a Wherein,representing the total depth of the three-dimensional bucket value;representing the total column number of the three-dimensional bucket numerical value;is a nodexIn thatkDepth layerjAn output weight estimate for the column; according to the nodexIn thatkDepth layerThe output weight estimated value of the column is obtained to obtain a nodexIn thatkThe output weight estimate for the depth layer is:(ii) a Wherein,is a nodexIn thatkAn output weight estimate for the depth layer; according to the nodexIn thatThe estimated value of the output weight of the depth is obtained to obtain a nodexThe node weight estimate of (a) is:(ii) a Wherein,is a nodexThe output weight estimate of (a);
for depthInIf each packet ofThen nodexIn thatkDepth layeriThe input weight estimates for the rows are:(ii) a Otherwise, the nodexIn thatkDepth layerjThe input weight estimates for the columns are:(ii) a Wherein,is a nodexIn thatkDepth layeriAn input weight estimate for the row; according to the nodexIn thatkDepth layerInput weight estimation value of the column to obtain a nodexIn thatkThe input weight estimate for the depth layer is:(ii) a Wherein,the total row number of the three-dimensional bucket numerical value is represented;is a nodexIn thatkAn input weight estimate for the depth layer; according to the nodexIn thatThe depth input weight estimated value is obtained to obtain a nodexThe node weight estimate of (a) is:(ii) a Wherein,is a nodexThe input weight estimate of (1); according to the nodexOutputting weight estimatesAnd input weight estimatesTo obtain a nodexThe node weight estimate of (1).
The bucket updating module 408 is further configured to perform statistics and query of stream data according to data stored in the three-dimensional sketch structure, where the sub-graph weight query includes: obtaining a subset of nodes of a subgraph to be queriedWhereinnRepresenting the number of nodes in the node subset;indicating a serial number ofiThe node(s) of (a) is (are),(ii) a Determining a subset of edges of a subgraph from nodes in the subset of nodesWhere m represents the number of edges in the subset of edges,;indicating a serial number ofiThe edge of (a) is provided with,(ii) a Obtaining weights of all edges in a subset of edges by an edge weight query(ii) a Will be provided withThe result of the sum of the weights of all edges inAs weight estimates for the subgraph.
The bucket updating module 408 is further configured to perform statistics and query on stream data according to data stored in the three-dimensional sketch structure, where the step of the path reachability query is: according to each bucket in the three-dimensional sketch structureSum of weighted values inSumming to obtain the total weight of the data stream in the current periodF(ii) a WhereinSequence numbers representing the three dimensions of row, column and depth of the bucket; traverse allWhen is coming into contact withAt the same time, readKey stored inObtained by edge weight queryThe edge weight of whenWhen it is determinedIs an elephant stream edge; whereinIs a preset threshold parameter; at all weights greater thanThe elephant stream edge carries out path reachability query through a breadth-first search algorithm, wherein the path reachability query is to determine whether the source node a can pass through the weight of at leastIs connected to the target node b.
For specific limitations of the design apparatus of the three-dimensional sketch structure, reference may be made to the above limitations of the design method of the three-dimensional sketch structure, which are not described herein again. All or part of each module in the design device of the three-dimensional sketch structure can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of designing a three-dimensional sketch structure. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for designing a three-dimensional sketch structure, the method comprising:
constructing a three-dimensional sketch structure for storing stream data information, wherein the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
dividing each bucket in the three-dimensional bucket array into three regions, respectively storing the sum of weight values mapped to all edges of the bucket, a key with the largest current weight mapped to the bucket and a value of an indication counter for indicating to reserve or replace the key with the largest current weight, and initializing the bucket;
acquiring stream data information of a graph structure, and calculating a bucket index of each edge at each depth according to node information of the edge in the stream data information and hash functions corresponding to different depths;
updating the bucket of each depth corresponding to the edge according to the bucket index, wherein the updating comprises: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and carrying out statistics and query on the stream data according to data stored in the three-dimensional sketch structure.
2. The method according to claim 1, wherein the bucket of each depth corresponding to the edge is updated according to the bucket index, and the updating comprises: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the maximum current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the maximum current weight according to the updated value of the indication counter, wherein the method comprises the following steps:
according toThe edge in the stream data informationThe bucket index ofUpdating the bucket of each depth corresponding to the edge, wherein,the depth sequence number of the packet,is composed ofThe hash function corresponding to the depth layer comprises the following steps:
updating the sum of the weight values in the bucket to be:
wherein,is the weight information of the edge in question,is an edgeMapped toThe sum of the weight values in bucket of the depth layer;
wherein,is an edgeMapped toThe key with the largest current weight in the bucket of the depth layer;is an edgeMapped toA value of the indication counter in a bucket of a depth layer;
judgment ofPositive and negative of (2)When the temperature of the water is higher than the set temperature,the value of (d) remains unchanged; when in useAt the same time, updateAndcomprises the following steps:
3. the method of claim 2, wherein the counting and querying of stream data according to the data stored in the three-dimensional sketch structure comprises:
and carrying out statistics and query of flow data according to data stored in the three-dimensional sketch structure, wherein the query comprises side weight query, node weight query, elephant flow side query, elephant flow node query, mutation flow query, mutation node query, subgraph weight query and path reachability query.
4. The method of claim 3, wherein the counting and querying of the stream data according to the data stored in the three-dimensional sketch structure comprises:
counting and inquiring flow data according to the data stored in the three-dimensional sketch structure, wherein the step of inquiring the edge weight is as follows:
Is compared to a depth ofkIn a bucket ofValue of (A) andwhether they are the same or not, if so, the edgeIn thatkThe edge weight estimate for the depth layer is:
according to the edgeThe minimum value of the edge weight estimated values at all depths is obtained to obtain the edgeThe edge weight estimate of (a) is:
5. The method of claim 3, wherein the counting and querying of the stream data according to the data stored in the three-dimensional sketch structure comprises:
and carrying out statistics and query on flow data according to the data stored in the three-dimensional sketch structure, wherein the node weight query step is as follows:
obtaining a node to queryx;
For depthMiddle row ofIf each packet ofThen nodexIn thatkDepth layerjThe output weight estimates for the columns are:
otherwise, the nodexIn thatkDepth layerjThe output weight estimates for the columns are:
wherein,representing the total depth of the three-dimensional bucket values;representing the total column number of the three-dimensional bucket numerical value;is a nodexIn thatkDepth layerjAn output weight estimate for the column;
according to the nodexIn thatkDepth layerThe output weight estimated value of the column is obtained to obtain a nodexIn thatkThe output weight estimate for the depth layer is:
according to the nodexIn thatThe estimated value of the output weight of the depth is obtained to obtain a nodexThe node weight estimate of (a) is:
for depthInIf each packet ofThen nodexIn thatkDepth layeriThe input weight estimates for the rows are:
otherwise, the nodexIn thatkDepth layerjThe input weight estimates for the columns are:
according to the nodexIn thatkDepth layerInput weight estimation value of the column to obtain a nodexIn thatkThe input weight estimate for the depth layer is:
wherein,the total row number of the three-dimensional bucket numerical value is represented;is a nodexIn thatkAn input weight estimate for the depth layer;
according to the nodexIn thatThe depth input weight estimated value is obtained to obtain a nodexThe node weight estimate of (a) is:
6. The method of claim 4, wherein the counting and querying of stream data according to the data stored in the three-dimensional sketch structure comprises:
performing statistics and query of stream data according to data stored in the three-dimensional sketch structure, wherein the sub-graph weight query step is as follows:
obtaining a subset of nodes of a subgraph to be queriedWhereinnRepresenting a number of nodes in the subset of nodes;indicating a serial number ofiThe node(s) of (a) is (are),;
determining a subset of edges of the subgraph from nodes in the subset of nodesWherein m represents the number of edges in the subset of edges,;indicating a serial number ofiThe edge of (a) is provided with,;
7. The method of claim 4, wherein the counting and querying of stream data according to the data stored in the three-dimensional sketch structure comprises:
and carrying out statistics and query on stream data according to the data stored in the three-dimensional sketch structure, wherein the step of path reachability query is as follows:
according to each bucket in the three-dimensional sketch structureSum of the weight values in (1)Summing to obtain the total weight of the data stream in the current periodF(ii) a WhereinSequence numbers representing the bucket in three dimensions of row, column and depth;
traverse allWhen is coming into contact withAt the same time, readKey stored inObtained by edge weight queryThe edge weight of whenWhen it is determinedIs an elephant stream edge; whereinIs a preset threshold parameter;
8. An apparatus for designing a three-dimensional sketch structure, the apparatus comprising:
the device comprises a three-dimensional sketch structure construction module, a data information storage module and a data information processing module, wherein the three-dimensional sketch structure construction module is used for constructing a three-dimensional sketch structure for storing stream data information, the three-dimensional sketch structure comprises a three-dimensional bucket array, the total line number and the total column number of the three-dimensional bucket array are the same, and different depths of the three-dimensional bucket array correspond to different hash functions;
a bucket initialization module, configured to divide each bucket in the three-dimensional bucket array into three regions, respectively store a sum of weight values mapped to all edges of the bucket, a key with a largest current weight mapped to the bucket, and a value of an indication counter used for indicating to reserve or replace the key with the largest current weight, and initialize the bucket;
the bucket index determining module is used for acquiring stream data information of a graph structure, and calculating a bucket index of each edge at each depth according to node information of the edge in the stream data information and hash functions corresponding to different depths;
a bucket updating module, configured to update the bucket of each depth corresponding to the edge according to the bucket index, where the updating includes: updating the sum of the weight values in the bucket according to the weight information of the edge, updating the value of the indication counter through a majority voting algorithm according to the key with the largest current weight in the bucket and the acquired information of the edge, and determining to reserve or replace the key with the largest current weight according to the updated value of the indication counter;
and the counting and inquiring module is used for continuously acquiring stream data information, updating information of all sides in the stream data information into the three-dimensional sketch structure, and counting and inquiring the stream data according to the data stored in the three-dimensional sketch structure.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110093365.3A CN112416950B (en) | 2021-01-25 | 2021-01-25 | Design method and device of three-dimensional sketch structure |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110093365.3A CN112416950B (en) | 2021-01-25 | 2021-01-25 | Design method and device of three-dimensional sketch structure |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112416950A true CN112416950A (en) | 2021-02-26 |
CN112416950B CN112416950B (en) | 2021-03-26 |
Family
ID=74782483
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110093365.3A Active CN112416950B (en) | 2021-01-25 | 2021-01-25 | Design method and device of three-dimensional sketch structure |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112416950B (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113225227A (en) * | 2021-03-25 | 2021-08-06 | 北京大学 | Network measurement method and device based on simplified diagram and considering simplicity and accuracy |
CN113670293A (en) * | 2021-08-11 | 2021-11-19 | 追觅创新科技(苏州)有限公司 | Map construction method and device |
CN113746700A (en) * | 2021-09-02 | 2021-12-03 | 中国人民解放军国防科技大学 | Elephant flow rapid detection method and system based on probability sampling |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100070509A1 (en) * | 2008-08-15 | 2010-03-18 | Kai Li | System And Method For High-Dimensional Similarity Search |
CN102855667A (en) * | 2011-06-30 | 2013-01-02 | 中国科学院深圳先进技术研究院 | Computer-assisted design system and method for three-dimensional objects |
CN106528815A (en) * | 2016-11-14 | 2017-03-22 | 中国人民解放军理工大学 | Method and system for probabilistic aggregation query of road network moving objects |
CN106951501A (en) * | 2017-03-16 | 2017-07-14 | 天津大学 | A kind of method for searching three-dimension model based on many figure matchings |
CN107077744A (en) * | 2014-10-03 | 2017-08-18 | 三星电子株式会社 | Generated using the threedimensional model at edge |
US20200106867A1 (en) * | 2018-12-03 | 2020-04-02 | Intel Corporation | Sketch Table For Traffic Profiling and Measurement |
-
2021
- 2021-01-25 CN CN202110093365.3A patent/CN112416950B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100070509A1 (en) * | 2008-08-15 | 2010-03-18 | Kai Li | System And Method For High-Dimensional Similarity Search |
CN102855667A (en) * | 2011-06-30 | 2013-01-02 | 中国科学院深圳先进技术研究院 | Computer-assisted design system and method for three-dimensional objects |
CN107077744A (en) * | 2014-10-03 | 2017-08-18 | 三星电子株式会社 | Generated using the threedimensional model at edge |
CN106528815A (en) * | 2016-11-14 | 2017-03-22 | 中国人民解放军理工大学 | Method and system for probabilistic aggregation query of road network moving objects |
CN106951501A (en) * | 2017-03-16 | 2017-07-14 | 天津大学 | A kind of method for searching three-dimension model based on many figure matchings |
US20200106867A1 (en) * | 2018-12-03 | 2020-04-02 | Intel Corporation | Sketch Table For Traffic Profiling and Measurement |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113225227A (en) * | 2021-03-25 | 2021-08-06 | 北京大学 | Network measurement method and device based on simplified diagram and considering simplicity and accuracy |
CN113225227B (en) * | 2021-03-25 | 2023-01-17 | 北京大学 | Network measurement method and device based on simplified diagram and considering simplicity and accuracy |
CN113670293A (en) * | 2021-08-11 | 2021-11-19 | 追觅创新科技(苏州)有限公司 | Map construction method and device |
CN113746700A (en) * | 2021-09-02 | 2021-12-03 | 中国人民解放军国防科技大学 | Elephant flow rapid detection method and system based on probability sampling |
Also Published As
Publication number | Publication date |
---|---|
CN112416950B (en) | 2021-03-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
To et al. | A framework for protecting worker location privacy in spatial crowdsourcing | |
CN112416950B (en) | Design method and device of three-dimensional sketch structure | |
Naor et al. | The load, capacity, and availability of quorum systems | |
Xiao et al. | A space-time diffusion scheme for peer-to-peer least-squares estimation | |
Wise et al. | Regionalisation tools for the exploratory spatial analysis of health data | |
Boguná et al. | Absence of epidemic threshold in scale-free networks with degree correlations | |
KR101999639B1 (en) | Data processing systems, compute nodes and data processing methods | |
Xuan et al. | Voronoi-based multi-level range search in mobile navigation | |
Zhao et al. | A blockchain-based approach for saving and tracking differential-privacy cost | |
Patgiri et al. | Role of bloom filter in big data research: A survey | |
Shirvani | Novel solutions and applications of the object partitioning problem | |
Cohen et al. | Spatially-decaying aggregation over a network | |
Ahmed et al. | Social graph publishing with privacy guarantees | |
Henry et al. | Practical approaches to varying network size in combinatorial key predistribution schemes | |
CN103200034B (en) | Network user structure disturbance method based on spectral constraint and sensitive area partition | |
Liu et al. | SEAD counter: Self-adaptive counters with different counting ranges | |
Chen et al. | Scube: Efficient summarization for skewed graph streams | |
Gao et al. | U 2-Tree: A Universal Two-Layer Distributed Indexing Scheme for Cloud Storage System | |
Liu et al. | Towards in-network compact representation: Mergeable counting bloom filter vis cuckoo scheduling | |
Chen et al. | A Sketch-based clustering algorithm for uncertain data streams | |
Kou et al. | A location privacy protection algorithm based on differential privacy in sensor network | |
Duan et al. | A secure online algorithm for link analysis on weighted graph | |
CN112765491B (en) | Link prediction method and device considering node local area link compactness | |
CN117609412B (en) | Spatial object association method and device based on network structure information | |
US20140258332A1 (en) | Fast distributed database frequency summarization |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |