CN113127491B - Flow graph dividing system based on correlation characteristics - Google Patents
Flow graph dividing system based on correlation characteristics Download PDFInfo
- Publication number
- CN113127491B CN113127491B CN202110468957.9A CN202110468957A CN113127491B CN 113127491 B CN113127491 B CN 113127491B CN 202110468957 A CN202110468957 A CN 202110468957A CN 113127491 B CN113127491 B CN 113127491B
- Authority
- CN
- China
- Prior art keywords
- data
- edge
- stream
- point
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Strategic Management (AREA)
- Computational Linguistics (AREA)
- Marketing (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a flow graph dividing system based on correlation characteristics, which comprises a data analysis module, a data rearrangement module, a point element storage module, an edge element storage module and a data navigation module. The data analysis module analyzes the transaction data stream into a data format of an associated map, and generates a point stream and an edge stream; the data rearrangement module carries out disorder rearrangement on the side stream data, and reduces the influence of specific transaction data on a subsequent division algorithm; the data navigation module selects a proper storage position for each side stream data; and the edge element storage module and the point element storage module write the divided edge flow data and point flow data into a database. The flow graph dividing system provided by the invention can optimize the partition process of the associated graph data according to the characteristics of the transaction data flow and improve the performance of the subsequent execution graph analysis task.
Description
Technical Field
The invention belongs to the field of transaction anti-fraud, and particularly relates to a flow graph dividing system based on correlation characteristics, which is suitable for distributed storage and analysis of flow transaction data.
Background
In the field of transaction anti-fraud, the structure of a data graph such as customer information, transaction records and the like is often modeled to construct a correlation map. For example, an association map may be constructed with bank card numbers as nodes and a transfer between bank cards as edges. And part of the bank card nodes can be marked as abnormal accounts, and analysis such as risk assessment and the like can be carried out on unmarked accounts based on the incidence relation expressed by the incidence map.
Common algorithms for associative graph analysis include graph traversal, community discovery, loop detection, connectivity detection, and the like. In practice, these analysis algorithms are typically implemented using a distributed graph computation framework. The mainstream graph computation framework adopts a Pregel-like message propagation model, and the computation complexity can be approximately represented by the communication quantity between distributed nodes. Therefore, by optimizing the dividing mode of the graph data, the communication load of the distributed graph calculation framework can be reduced, and the overall calculation performance is improved.
In real world applications, large amounts of data are flooded into the system in a data stream. For example, a single transaction is taken as a piece of data, which includes attributes of debit, credit, time, platform, geographic location, etc. The system constructs these data into a graph according to preset meta-rules, for example, defining the borrower and the lender as two nodes on the graph, and creating an edge between the two nodes, wherein the nodes and the edge record other information of the transaction in an attribute mode. Typically, the node attributes include information that is fixed and unchangeable for the transaction entity, such as a bank card number, an account opening bank address, an account opening mobile phone number, an identification number, etc., while the edge attributes include information specific to a single transaction, such as time, platform, amount of money, etc.
As can be seen from the above example, the transaction data stream is large in size and complex in format. The problem that the association graph is constructed in real time by transaction data flow, the graph is guaranteed to be efficiently divided as far as possible while the load balance of storage and calculation nodes is guaranteed is relatively difficult.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a flow graph dividing system for constructing an associated graph facing transaction data flow, which is operated before an associated graph analysis task, realizes the real-time partition of the associated graph and ensures the load balance of nodes.
The purpose of the invention is realized by the following technical scheme: a flow graph dividing system based on correlation characteristics comprises a data analysis module, a data rearrangement module, a point element storage module, an edge element storage module and a data navigation module.
The data analysis module is used for analyzing the original transaction data stream received by the system and generating a point data stream and an edge data stream of the associated map, which are called point stream and edge stream for short. Specifically, the format of point data is defined through meta-rules, and the point data comprises a main attribute and a non-main attribute; defining the format of edge data through meta-rules, wherein the edge data comprises main attributes of two endpoints and non-main attributes of edges; the main attribute is used as the unique identification of the point, and the non-main attribute is used as the attribute description of the point. The generated dot flow data and edge flow data of the original transaction data stream are transmitted to a data rearrangement module.
The data rearrangement module is used for disturbing the edge flow data according to a certain rule and reducing the interference of the specific transaction data flow sequence to the flow chart division algorithm. The data rearrangement module provides a preset size ofThe data accumulation queue of (1) stores the side stream data sent from the upstream data analysis module into the end of the data accumulation queue first, and then exchanges with a certain data in the queue randomly. Each side stream data comprises a time stamp for recording the time when the side stream data enters the queue, and the stay time of the side stream data in the queue exceeds a preset valueAnd then pushed directly to the downstream data navigation module. When the size of the data accumulation queue exceeds the preset sizeAnd in time, the head of the queue data is pushed to a downstream data navigation module. Only the edge stream data enters the data accumulation queue and is finally led into the data navigation module, and the point stream data is directly pushed to the point meta-storage module.
And the data navigation module is used for determining the specific storage position of the side stream data. Defining an overall system comprisingThe edge cell storage partition is used for storing edge stream data and is numbered as. For each partitionMaintaining a local bloom filterFor recording whether the partition contains an end point of certain side stream data. First, thejThe stream data of the edge is recorded as,Andare respectively the firstjThe primary attributes of the start and end points of the stream of edge streams,are respectively the firstjThe non-primary attributes of the starting and ending points of the stream of edge streams,is the firstjNon-primary property of an edge stream data edge. To partitionDesign of an objective functionDividing endpoints of the edge stream data with the associated characteristics into the same partition; the partition is selected so that the value of the objective function is maximized, i.e. the partition is numberedAnd will beThe stream data of the edgePush to partitionThe edge cell storage module. If there are multiple maximums at the same timeThen one is randomly selected.
The point element storage module is used for storing the point flow data into a global key-value lookup table. For a primary attribute ofOf the point stream data ofAnd writing the main attribute and the non-main attribute of the dot flow data into the HBase database as a main key. If there are duplicate non-primary attributes, the latest version is retained.
The edge storage module is used for storing the edge stream data into the local key-value lookup table. For the starting pointTerminal pointTo side stream data ofAs a primary key HBase database.Indicating strings of arbitrary lengthConversion to a fixed length ofBy default, of a character stringIs the maximum length that can represent the node's primary attribute.The method is a character string splicing operation. HBase according toAnd the edge element storage partition regions and appoint the regions for storage according to the partition result of the data navigation module.
Furthermore, in the data analysis module, each transaction datum is a key-value dictionary table represented by a json format, and various information when a transaction occurs is recorded in detail.
Further, the data analysis module generates the dot stream data and the edge stream data according to a predefined meta-rule as follows:
a) for each defined point data format in the meta-rule, its primary attribute isChecking whether the transaction data includesA field, if contained, generating a main attribute value as transaction dataThe stream of points of value and into the stream of points of data. The non-main attribute of the point is defined according to the meta-rule, is obtained from the transaction data, and is ignored if the non-main attribute does not exist.
b) For each defined edge data in a meta-rule, the main attributes of its two endpoints areAndchecking whether the transaction data includesA field andand a field, only two fields are contained at the same time, one side stream data is generated, and the side stream is pushed. Other attributes of the side-stream data are defined according to meta-rules, are obtained from the transaction data, and are ignored if not present.
c) For a single transaction datum, the data can be analyzed into a plurality of flow chart data such as a single point, two points or one side of two points according to the specific definition of the meta rule.
Further, the data navigation module maintains two globally distributed key-value storage structuresAnd。is a distributed hash table for storing a mapping of an arbitrary string to a 64-bit positive integer value.Is a global bloom filter for determining whether any string exists.Andredis implementation deployed by Cluster mode.
Further, the data navigation module, its partitioned objective functionConsists of two parts. The first part punishs the unbalanced data division by the specific formulaWhereinAndto be hyper-parametric, partitionThe amount of stored data isMaximum data capacity per partition ofMinimum data capacity of. The second part is used for optimizing the locality of data division and has the specific formula. Wherein the function。Is a bloom filter, when partitionedPoint of presenceWhen the temperature of the water is higher than the set temperature,otherwise return to。Andin order to be a hyper-parameter,for nodes in a written flow graphDegree of (c).For evaluating node non-primary attribute setsAnd partitioningThe degree of matching of (1), specifically, each non-primary attribute in the setCalculating,The operator is spliced for the string.For a global bloom filter, return when incoming parameters existOtherwise return to。Is a distributed hash table that disregards key collisions. Finally obtaining,。
Further, in the edge storage module, for the repeated edges, a service-related aggregation function may be adopted to combine the attributes of the edges, so as to reduce the storage space.
The invention has the following beneficial effects: aiming at the spatial and temporal locality of transaction entities, transaction data with similar characteristics can be preferentially stored in the same partition, and point data copy during the operation of an offline analysis task is reduced, so that the communication load is reduced, and the overall operation efficiency is improved. The invention can determine the storage partition of each transaction record in constant time, is irrelevant to the size of a graph data set, can construct an associated graph for a transaction data stream in real time, and writes the associated graph into a storage module according to the requirement of graph partition optimization.
Drawings
FIG. 1 is a block diagram of the system;
FIG. 2 is a data navigation module service flow diagram.
Detailed Description
The following describes embodiments of the present invention in further detail with reference to the accompanying drawings.
As shown in fig. 1, the invention relates to a flow graph partitioning system based on association features, wherein the flow graph partitioning system automatically partitions a finally generated association graph within a sub-linear time complexity under the condition that only each transaction data and limited statistical information can be seen, and simultaneously meets the requirements of load balancing, reduction of communication load of subsequent tasks and the like. The system comprises a data analysis module, a data rearrangement module, a point element storage module, an edge element storage module and a data navigation module.
The data analysis module is used for analyzing the original transaction data stream received by the system, each transaction data is a key-value dictionary table represented by a json format, and various information when a transaction occurs is recorded in detail. And generating a point data stream and an edge data stream of the associated map according to the original transaction data stream, which are referred to as the point stream and the edge stream for short. Specifically, the format of point data is defined through meta-rules, and the point data comprises a main attribute and a non-main attribute; defining the format of edge data through meta-rules, wherein the edge data comprises main attributes of two endpoints and non-main attributes of edges; the main attribute is used as the unique identification of the point, and the non-main attribute is used as the attribute description of the point. The generated dot flow data and edge flow data of the original transaction data stream are transmitted to a data rearrangement module. The data analysis module generates point stream data and edge stream data according to a predefined meta-rule as follows:
a) for each defined point data format in the meta-rule, its primary attribute isChecking whether the transaction data includesA field, if contained, generating a main attribute value as transaction dataThe stream of points of value and into the stream of points of data. The non-main attribute of the point is defined according to the meta-rule, is obtained from the transaction data, and is ignored if the non-main attribute does not exist.
b) For each defined edge data in a meta-rule, the main attributes of its two endpoints areAndchecking whether the transaction data includesA field andand a field, only two fields are contained at the same time, one side stream data is generated, and the side stream is pushed. Other attributes of the side-stream data are defined according to meta-rules, are obtained from the transaction data, and are ignored if not present.
c) For a single transaction datum, the data can be analyzed into a plurality of flow chart data such as a single point, two points or one side of two points according to the specific definition of the meta rule.
The data rearrangement module is used for disturbing the edge flow data according to a certain rule and reducing the interference of the specific transaction data flow sequence to the flow chart division algorithm. The data rearrangement module provides a preset size ofThe data accumulation queue of (1) stores the side stream data sent from the upstream data analysis module into the end of the data accumulation queue first, and then exchanges with a certain data in the queue randomly. Each side stream data comprises a time stamp for recording the time when the side stream data enters the queue, and the stay time of the side stream data in the queue exceeds a preset valueAnd then pushed directly to the downstream data navigation module. When the size of the data accumulation queue exceeds the preset sizeAnd in time, the head of the queue data is pushed to a downstream data navigation module. Only the edge stream data enters the data accumulation queue and is finally led into the data navigation module, and the point stream data is directly pushed to the point meta-storage module.
As shown in fig. 2, the data navigation module is configured to determine a specific storage location of the side stream data. Defining an overall system comprisingThe edge cell storage partition is used for storing edge stream data and is numbered as. For each partitionMaintaining a local bloom filterFor recording whether the partition contains an end point of certain side stream data. First, thejThe stream data of the edge is recorded as,Andare respectively the firstjThe primary attributes of the start and end points of the stream of edge streams,are respectively the firstjThe non-primary attributes of the starting and ending points of the stream of edge streams,is the firstjNon-primary property of an edge stream data edge. To partitionDesign of an objective functionDividing endpoints of the edge stream data with the associated characteristics into the same partition; the partition is selected so that the value of the objective function is maximized, i.e. the partition is numberedAnd stream the stream of the edge streamPush to partitionThe edge cell storage module. If there are multiple maximums at the same timeThen one is randomly selected.
Data navigation module maintains two globally distributed key-value storage structuresAnd。is a distributed hash table for storing a mapping of an arbitrary string to a 64-bit positive integer value.Is a global bloom filter for determining whether any string exists.Andredis implementation deployed by Cluster mode.
The data navigation module, the partitioned target function thereofConsists of two parts. The first part punishs the unbalanced data division by the specific formulaWhereinAndto be hyper-parametric, partitionThe amount of stored data isMaximum data capacity per partition ofMinimum data capacity of. The second part is used for optimizing the locality of data division and has the specific formula. Wherein the function。Is a bloom filter, when partitionedPoint of presenceWhen the temperature of the water is higher than the set temperature,otherwise return to。Andin order to be a hyper-parameter,for nodes in a written flow graphDegree of (c).For evaluating node non-primary attribute setsAnd partitioningThe degree of matching of (1), specifically, each non-primary attribute in the setCalculating,The operator is spliced for the string.For a global bloom filter, return when incoming parameters existOtherwise return to。Is a distributed hash table that disregards key collisions. Finally obtaining,。
The point element storage module is used for storing the point flow data into a global key-value lookup table. For a primary attribute ofOf the point stream data ofAnd writing the main attribute and the non-main attribute of the dot flow data into the HBase database as a main key. If there are duplicate non-primary attributes, the latest version is retained.
The edge storage module is used for storing the edge stream data into the local key-value lookup table. For the starting pointTerminal pointTo side stream data ofAs a primary key HBase database.Indicating strings of arbitrary lengthConversion to a fixed length ofBy default, of a character stringIs the maximum length that can represent the node's primary attribute.The method is a character string splicing operation. HBase according toAnd the edge element storage partition regions and appoint the regions for storage according to the partition result of the data navigation module. For repeated edges, a business-related aggregation function may be employed to combine the attributes of the edges to reduce storage space.
Example (b):
the invention provides a system for constructing an association map from transaction data flow, dividing a flow graph in real time and finally writing the association map into a bottom database. According to the module sequence, the whole process atmosphere comprises three steps: a) analyzing the data and generating an out-of-order point flow and an edge flow; b) dividing the edge stream data and generating the edge stream data with partition marks; c) and generating a primary key for the point flow and the edge flow, and writing the primary key into an HBase database.
An example is given below for three steps, respectively:
step a) analyzing data and generating an out-of-order point flow and an edge flow:
declaring a transaction datum expressed in json format as follows:
{
"borrower" includes a first opening
“id”: 0001,
13511112222 is used as a mobile phone, and the mobile phone is provided with a mobile phone cover,
a card number of 123456789,
"Account opening website" xxx01 "
},
"lending side" for containing Chinese dictionary
“id”: 0002,
13522221111 is used as a mobile phone, and the mobile phone is provided with a mobile phone cover,
card number 987654321
},
"transaction amount": 100.00 ",
"trading platform": dd ",
123999923212 for transaction time
}
The system administrator defines the metadata containing point structure as:
{
entrance field [ "borrower", "lender" ]
"Main attribute": id ",
non-main attribute [ "card number", "account opening site" ]
}
The edge structure is:
{
primary property
"origin" means "borrower",
"end point": lender "
},
Non-primary attribute [ "transaction amount", "transaction time" ]
}
As described by the data parsing module, the system parses the raw transaction data into blob flow data and edge flow data based on the metadata. As illustrated by example, the stream of point-of-flow data contains= 0001, card number 123456789, account opening site xxx01 and= { "id": 0002, "card number": 987654321}, and the side stream data comprises= starting point { "id": 0001}, "ending point {" id ": 0002}," transaction amount ": 100.00 {" transaction amount { "start point": 0001}, "" end point ": 0002}," "transaction amount": 100.00 { "transaction amount {" start point ": and" } end point "{" end point ": 5 {" end point ": 1 {" end point ": and" { "end point": 5 { "end point": and ": 5 {" end point ": 1 {" end point ": and" } end point ": 100."Transaction time': 123999923212 }.
The point flow data obtained by analysis can be directly sent to a downstream point element storage module and immediately written into an HBase database. The parsed edge stream data is sent to a data rearrangement module, and is sent to a data navigation module after waiting for a limited time.
And b) dividing the boundary stream data to obtain boundary stream data with partition marks. For number ofThe target function is a partition selection function, and the partition selection function is calculated. As has been described in the examples herein,is the starting point master attribute, with a value of 0001,is the endpoint primary attribute, value 0002;is a list of attributes for the starting point described above,is the above list of endpoint attributes. Wherein,Andfor the preset maximum capacity and minimum capacity of the partition,,andcalculated according to the following formula:
whereinAre all the super-parameters of the system,is a string concatenation function.Andare functions for determining the presence or absence of strings, whereinPrimarily for determining node dominance such asWhether or not it is in a partitionIn the above-mentioned manner,the method is used for judging whether any character string appears, and the two functions are realized by adopting a bloom filter. WhereinThe occurrence frequency of the character string is mainly returned, and the method is realized by using a distributed hash table. Suppose thatProperty ofXxx01, the account opening websiteIs "3 @ point of opening @ xxx 01". When the character stringAppeared out of dateReturn 1, otherwise return 0. Each time a string appears in the new dataThen, then. In view of the performance requirements,key conflicts can be resolved without means such as a linked list and the like, and certain data errors are allowed.Should be designed according to the attributes of the traffic field,for example, when one attribute is "location": street yy in xx area of hangzhou, zhejiang, the attribute granularity can be adjusted to "location": street of hangzhou, zhejiang, and then generalized hash value calculation is performed.
As shown in fig. 2, the main flow of step b) is to sequentially calculate the partition selection functionThe value on each partition is then selected, the partition with the largest valueAs the target partition for the data.
And c) generating a main key for the point stream and the edge stream, and writing the main key into an HBase database. The primary key of the stream of point flow isAndi.e. 0001 and 0002 as described in the above examples. The main key of the side stream data is designed asAssuming that the length of data required to store partition encoding is 3, the length required to store the main attribute of stream data is 5,=3, then the above example describes=003@00001@ 00002. When creating HBase table, should be based onPartitioning regions by the highest 3 bits to ensure identityCan be written to the same batch of regions.
The above-described embodiments are intended to illustrate rather than to limit the invention, and any modifications and variations of the present invention are within the spirit of the invention and the scope of the appended claims.
Claims (6)
1. A flow graph dividing system based on correlation characteristics is characterized by comprising a data analysis module, a data rearrangement module, a point element storage module, an edge element storage module and a data navigation module;
the data analysis module is used for analyzing the original transaction data stream received by the system and generating a point data stream and an edge data stream of the associated map, which are called point stream and edge stream for short; specifically, the format of point data is defined through meta-rules, and the point data comprises a main attribute and a non-main attribute; defining the format of edge data through meta-rules, wherein the edge data comprises main attributes of two endpoints and non-main attributes of edges; the main attribute is used as the unique identification of the point, and the non-main attribute is used as the attribute description of the point; point flow data and edge flow data generated by the original transaction data flow are transmitted to a data rearrangement module;
the data rearrangement module is used for disturbing the edge flow data according to a certain rule and reducing the interference of the sequence of the specific transaction data flow to the flow chart division algorithm; the data rearrangement module provides a preset size ofThe data accumulation queue, the side stream data sent from the upstream data analysis module is firstly stored at the end of the data accumulation queue, and then is randomly exchanged with a certain data in the queue; each side stream data comprises a time stamp for recording the time when the side stream data enters the queue, and the stay time of the side stream data in the queue exceeds a preset valueThen the data is directly pushed to a downstream data navigation module; when the size of the data accumulation queue exceeds the preset sizeIn time, the head data of the queue is pushed to a downstream data navigation module; only the edge stream data enters a data accumulation queue and is finally led into a data navigation module, and the point stream data is directly pushed to a point element storage module;
the data navigation module is used for determining the specific storage position of the side stream data; defining an overall system comprisingThe edge cell storage partition is used for storing edge stream data and is numbered as(ii) a For each partitionMaintaining a local bloom filterAn endpoint for recording whether the partition contains certain edge stream data; first, thejThe stream data of the edge is recorded as,Andare respectively the firstjThe primary attributes of the start and end points of the stream of edge streams,are respectively the firstjThe non-primary attributes of the starting and ending points of the stream of edge streams,is the firstjNon-primary property of the edge stream data edge; to partitionDesign of an objective functionDividing endpoints of the edge stream data with the associated characteristics into the same partition; the partition is selected so that the value of the objective function is maximized, i.e. the partition is numberedAnd stream the stream of the edge streamPush to partitionThe edge element storage module; if there are multiple maximums at the same timeThen one is randomly selected;
the point element storage module is used for storing point stream data into a global key-value lookup table; for a primary attribute ofOf the point stream data ofWriting the main attribute and the non-main attribute of the dot stream data into an HBase database as a main key; if there are duplicate non-primary attributes, then the latest version is retained;
the edge storage module is used for storing the edge stream data into a local key-value lookup table; for the starting pointTerminal pointTo side stream data ofAs a primary key HBase database;indicating strings of arbitrary lengthConversion to a fixed length ofBy default, of a character stringThe maximum length can represent the main attribute of the node;performing character string splicing operation; HBase according toAnd the edge element storage partition regions and appoint the regions for storage according to the partition result of the data navigation module.
2. The system for dividing a flow graph based on associated features of claim 1, wherein in the data parsing module, each transaction datum is a key-value dictionary table represented by json format, and various information when a transaction occurs is recorded in detail.
3. The system for dividing a flow graph based on associated features according to claim 1, wherein the data parsing module generates the dot flow data and the edge flow data according to a predefined meta-rule as follows:
a) for each defined point data format in meta-rule with main attribute m, checking whether transaction data contains dataA field, if contained, generating a main attribute value as transaction data(ii) a stream of point data of values and pushing in the stream of point data; the non-main attribute of the point is defined according to the meta-rule, is obtained from the transaction data, and is ignored if the non-main attribute does not exist;
b) for each defined edge data in a meta-rule, the main attributes of its two endpoints areAndchecking whether the transaction data includesA field anda field, generating an edge stream data only when two fields are contained simultaneously, and pushing the edge stream; other attributes of the side stream data are defined according to meta-rules, obtained from the transaction data, and if the attributes do not exist, the attributes are ignored;
c) for a single transaction, the data is parsed into single point, two point or two point one-edge flow graph data according to the specific definition of the meta-rule.
4. The system for flow graph partitioning based on associative features according to claim 1, wherein the data navigation module maintains two globally distributed key-value storage structuresAnd;is a distributed hash table for storing the mapping from any string to a 64-bit positive integer value;is a global bloom filter for judging whether any string exists;andredis implementation deployed by Cluster mode.
5. The correlation-feature-based flow graph partitioning system according to claim 4, wherein the data navigation module is a partitioned objective functionConsists of two parts; the first part punishs the unbalanced data division by the specific formulaWhereinAndto be hyper-parametric, partitionStored numberAccording to the quantity ofMaximum data capacity per partition ofMinimum data capacity of(ii) a The second part is used for optimizing the locality of data division and has the specific formula(ii) a Wherein the function;Is a bloom filter, when partitionedPoint of presenceWhen the temperature of the water is higher than the set temperature,otherwise return to;Andin order to be a hyper-parameter,for nodes in a written flow graphDegree of (d);for evaluating node non-primary attribute setsAnd partitioningThe degree of matching of (1), specifically, each non-primary attribute in the setCalculating,Splicing operators for the character strings;for a global bloom filter, return when incoming parameters existOtherwise return to;Is a distributed hash table; finally obtaining,。
6. The system according to claim 1, wherein for the repeated edges, a service-dependent aggregation function is adopted to combine attributes of the edges in the edge storage module to reduce storage space.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110468957.9A CN113127491B (en) | 2021-04-28 | 2021-04-28 | Flow graph dividing system based on correlation characteristics |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110468957.9A CN113127491B (en) | 2021-04-28 | 2021-04-28 | Flow graph dividing system based on correlation characteristics |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113127491A CN113127491A (en) | 2021-07-16 |
CN113127491B true CN113127491B (en) | 2022-03-22 |
Family
ID=76780928
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110468957.9A Active CN113127491B (en) | 2021-04-28 | 2021-04-28 | Flow graph dividing system based on correlation characteristics |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113127491B (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7689602B1 (en) * | 2005-07-20 | 2010-03-30 | Bakbone Software, Inc. | Method of creating hierarchical indices for a distributed object system |
US8972337B1 (en) * | 2013-02-21 | 2015-03-03 | Amazon Technologies, Inc. | Efficient query processing in columnar databases using bloom filters |
CN109426574A (en) * | 2017-08-31 | 2019-03-05 | 华为技术有限公司 | Distributed computing system, data transmission method and device in distributed computing system |
CN109740037A (en) * | 2019-01-02 | 2019-05-10 | 山东省科学院情报研究所 | The distributed online real-time processing method of multi-source, isomery fluidised form big data and system |
CN110704630A (en) * | 2019-04-15 | 2020-01-17 | 中国石油大学(华东) | Self-optimization mechanism for identified associated graph |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11636154B2 (en) * | 2019-09-26 | 2023-04-25 | Fungible, Inc. | Data flow graph-driven analytics platform using data processing units having hardware accelerators |
US11636115B2 (en) * | 2019-09-26 | 2023-04-25 | Fungible, Inc. | Query processing using data processing units having DFA/NFA hardware accelerators |
-
2021
- 2021-04-28 CN CN202110468957.9A patent/CN113127491B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7689602B1 (en) * | 2005-07-20 | 2010-03-30 | Bakbone Software, Inc. | Method of creating hierarchical indices for a distributed object system |
US8972337B1 (en) * | 2013-02-21 | 2015-03-03 | Amazon Technologies, Inc. | Efficient query processing in columnar databases using bloom filters |
CN109426574A (en) * | 2017-08-31 | 2019-03-05 | 华为技术有限公司 | Distributed computing system, data transmission method and device in distributed computing system |
CN109740037A (en) * | 2019-01-02 | 2019-05-10 | 山东省科学院情报研究所 | The distributed online real-time processing method of multi-source, isomery fluidised form big data and system |
CN110704630A (en) * | 2019-04-15 | 2020-01-17 | 中国石油大学(华东) | Self-optimization mechanism for identified associated graph |
Non-Patent Citations (1)
Title |
---|
基于互联网的物理对象多域协同感知与融合分析技术;张毅;《中国优秀博硕士学位论文全文数据库(硕士)》;20190115(第01期);全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN113127491A (en) | 2021-07-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2016004813A1 (en) | Data storage method, query method and device | |
US12001800B2 (en) | Semantic-aware feature engineering | |
US10102536B1 (en) | Micro-geographic aggregation system | |
US8706758B2 (en) | Flexible account reconciliation | |
US20210182859A1 (en) | System And Method For Modifying An Existing Anti-Money Laundering Rule By Reducing False Alerts | |
CN106776848B (en) | Database query method and device | |
CN109829721B (en) | Online transaction multi-subject behavior modeling method based on heterogeneous network characterization learning | |
CN111090780A (en) | Method and device for determining suspicious transaction information, storage medium and electronic equipment | |
CN110728301A (en) | Credit scoring method, device, terminal and storage medium for individual user | |
WO2023165271A1 (en) | Knowledge graph construction and graph calculation | |
CN110349013A (en) | Risk control method and device | |
CN115687432A (en) | Method, apparatus, and medium for monitoring anomalous transaction data | |
US11188981B1 (en) | Identifying matching transfer transactions | |
CN106844541B (en) | Online analysis processing method and device | |
CN113127491B (en) | Flow graph dividing system based on correlation characteristics | |
CN110363534B (en) | Method and device for identifying abnormal transaction | |
CN112348657A (en) | Method and device for determining target credit user, computer equipment and storage medium | |
EP4170574A1 (en) | Synthesizing user transactional data for de-identifying sensitive information | |
CN116611914A (en) | Salary prediction method and device based on grouping statistics | |
CN114331665A (en) | Training method and device for credit judgment model of predetermined applicant and electronic equipment | |
CN112347102B (en) | Multi-table splicing method and multi-table splicing device | |
CN109919811B (en) | Insurance agent culture scheme generation method based on big data and related equipment | |
CN116954591B (en) | Generalized linear model training method, device, equipment and medium in banking field | |
US20220107949A1 (en) | Method of optimizing search system | |
CN116611923A (en) | Knowledge graph-based risk data acquisition method, system, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |