CN112702264A - Distributed network feature calculation method - Google Patents
Distributed network feature calculation method Download PDFInfo
- Publication number
- CN112702264A CN112702264A CN202011357397.1A CN202011357397A CN112702264A CN 112702264 A CN112702264 A CN 112702264A CN 202011357397 A CN202011357397 A CN 202011357397A CN 112702264 A CN112702264 A CN 112702264A
- Authority
- CN
- China
- Prior art keywords
- storage unit
- structure storage
- characteristic
- calculation
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 40
- 230000014509 gene expression Effects 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims description 9
- 230000006855 networking Effects 0.000 claims description 3
- 238000011161 development Methods 0.000 abstract description 4
- 238000013461 design Methods 0.000 abstract description 3
- 238000012423 maintenance Methods 0.000 description 2
- 238000010195 expression analysis Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003997 social interaction Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/44—Distributed routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/6245—Modifications to standard FIFO or LIFO
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed network characteristic calculation method, which comprises the following steps: generating and analyzing a characteristic expression and calculating a distributed route, wherein the characteristic expression is generated and analyzed by extracting elements in the characteristic expression to form an analyzable list; analyzing the feature expression according to the analyzable list; the distributed routing computation includes: and distributing a social network to be calculated for each node according to a weight algorithm, and after each node receives the calculation characteristic message, pulling the network distributed to the node by each node to perform characteristic calculation. The invention is based on the analysis and distributed computation of the complex characteristic expression, adapts the possibility of complex computation of all characteristic conditions in a service scene, improves the development efficiency in a template design mode, and greatly shortens the characteristic computation time in a distributed computation mode.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a distributed network feature calculation method.
Background
In services such as wind control, social interaction, e-commerce, etc., data of a relationship network is often used. For example, if you buy mother-infant products from a relatively close person in the internet where you are located in the e-commerce, then you can recommend such products to increase the purchase rate. For example, in financial wind control, the number of intermediaries and electronic fraud people in the network where you are located is large, the relationship network where you are located is a high-risk network, and loan applications for people inside the network are rejected with a high probability, so that risks are reduced. Therefore, the network characteristics of the object are important indicators for extending the image of the object. In the anti-fraud business scene, a relationship network of a person is often analyzed, and the results of the characteristics can be used as important indexes for judging risk personnel, such as the proportion of overdue people in the current relationship network, the number of people in a GPS cell, the average credit line in the network, and the like. However, in a real business scenario, a network is often configured with hundreds of features, which means that a user's request is performed hundreds of times on the network feature.
At present, most of network characteristic calculation is carried out by adopting a graph database method, a graph database environment is built according to official documents, relevant data is imported, and statistics is carried out on the network in a database according to configured characteristics and the graph database is maintained. However, this method requires deployment and maintenance of a highly available graph database environment, the development and maintenance costs are high, and the learning cost is high due to the query language for which the business side needs to learn the graph database. Meanwhile, certain whole network data cannot be queried quickly and accurately, and the network characteristic calculation speed is low.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a distributed network characteristic calculation method, which aims to: based on the analysis of the complex feature expression and the distributed computation, the method adapts the possibility of complex computation of all feature conditions in a service scene, improves the development efficiency in a template design mode, and greatly shortens the feature computation time in a distributed computation mode.
The invention relates to a distributed network characteristic calculation method, which comprises the steps of characteristic expression generation and analysis and distributed route calculation,
the characteristic expression generation and analysis comprises the following steps: extracting elements in the characteristic expression to form an element queue; establishing a list structure storage unit and a stack structure storage unit; popping all the remaining elements in the stack structure storage unit out of the stack structure storage unit according to a last-in first-out sequence, and sequentially putting the elements into the list structure storage unit to form an analyzable list; and analyzing the characteristic expression.
Further, the generating and analyzing the feature expression further includes:
dividing elements in the feature expression into three types of variables, operators and separators, establishing a queue structure storage unit in a memory of the feature analysis device, and extracting and storing all elements in the feature expression into the queue structure storage unit one by one according to a left-to-right sequence to form an element queue;
establishing a list structure storage unit and a stack structure storage unit; traversing the element queue, sequentially placing variables in the element queue into a list structure storage unit, sequentially placing an operator and a separator into a stack structure storage unit, and sequentially popping the separator and a related operator from the stack structure storage unit and placing the separator and the related operator into the list structure storage unit according to the separator of the operation priority attribute placed in the stack structure storage unit;
after traversing the element queue, popping all the remaining elements in the stack structure storage unit out of the stack structure storage unit according to a last-in first-out sequence, and sequentially putting the elements into the list structure storage unit, thereby forming an analyzable list;
traversing the analyzable list, sequentially putting elements in the analyzable list into a stack structure storage unit, if the put elements are operators, popping variables related to the operators in the stack structure storage unit from the stack structure storage unit respectively and operating the variables with the operators, and putting the operated result into the stack structure storage unit until no element exists in the stack structure storage unit.
The invention greatly enriches the diversity of the feature configuration by analyzing and calculating the complex feature expression and provides more strategy configurations for the service party.
The distributed routing computation includes: after the networking of the social network is finished, the data of the network ID, the nodes and the edges are saved; inquiring nodes participating in feature calculation and weight values of corresponding nodes; and distributing the social network to be calculated for the nodes according to the weight algorithm, storing the social network to be calculated in a redis database, broadcasting and notifying the node machines participating in calculation, and pulling the social network distributed to each node for characteristic calculation after the node machines receive the information of calculating the characteristic task. The weighting algorithm is a polling algorithm to be weighted, and assuming 3 machines A, B, C, the default weight of each machine is 1. When distributing network data, the network data are distributed according to the same weight A B C A B C. If the weight of A is now 2, then it is de-assigned according to A A B C.
Further, the distributed routing computation further comprises:
each node machine participating in calculation is stored in a node detail table of the database, the node detail table comprises a weight value of each node machine, when a social network to be calculated is distributed, the weight value of each node machine is used as a weight proportion to carry out weighting calculation, and a calculated result is stored in a redis with a host name of key;
according to the published subscription message carried by the redis, each node machine participating in the calculation can subscribe a fixed message of 'calculation characteristic task', when the message is published, the node subscribing the message can receive the message data, and after the message is received, each calculation node machine pulls a corresponding standby calculation network from the redis through the host name key to perform characteristic calculation.
The distributed routing strategy can load tens of millions of characteristics to different computing nodes, greatly shortens the time of characteristic computation, supports the horizontal expansion of the computing nodes, and can dynamically distribute the characteristic quantity to the computing nodes with different performances through the configuration of the weight value.
The invention is composed of the feature expression generation and analysis and the distributed route calculation, although there is no logic relation between the feature expression generation and analysis and the distributed route calculation, the two must be possessed together to realize the purpose of the invention: based on the analysis of the complex expression and the distributed computation, the possibility of complex computation of all characteristic conditions in a service scene is adapted, the development efficiency is improved in a template design mode, and the time of the characteristic computation is greatly shortened in a distributed computation mode.
Drawings
The invention will now be described, by way of example, with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a feature expression generation algorithm of a distributed network feature computation method of the present invention;
FIG. 2 is a flow chart of a feature expression parsing algorithm of a distributed network feature computation method of the present invention;
FIG. 3 is a flow chart of distributed route computation of a distributed network feature computation method of the present invention;
FIG. 4 is a table of node details in a database;
FIG. 5 is a graph of network data to be calculated based on the weight assignment configured for each machine;
fig. 6 is a flow chart of a broadcast notification node machine performing computations.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
In the description of the embodiments of the present application, it should be noted that the terms "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings or orientations or positional relationships that the products of the present invention are usually placed in when used, and are only used for convenience of description and simplicity of description, but do not indicate or imply that the devices or elements that are referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present application. Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
The present invention will be described in detail with reference to fig. 1, 2 and 3.
1. As shown in fig. 1, a flow chart of a feature expression generation and analysis algorithm of a distributed network feature calculation method of the present invention is shown, where the feature expression generation algorithm includes:
many features are configured for each social network, and the features are actually arithmetic expressions in nature, for example, "the ratio of overdue number in network" is "overdue number/total number in network". For the sake of understanding, we will exemplify a basic arithmetic expression.
First an expression is divided into three parts, variables (e.g. 3,0, 2, etc.), operators (e.g. +, -,. one, etc.), separators (braces, etc.)
Declaring a basic expression such as 3+ (3-0) × 2;
traversing the declared expressions generates the underlying lists tokens ("3", "+", "(", "3", "-", "0", ")", "2");
declaring a list and a stack; when the traversal list tokens meets the variable, the variable is put into the list tokens, and when the operator is met, the stack is pressed;
such as: when a variable 3 is met, adding a list (3); encountering operator +, push Stack (+); encountering a separator "(", pushing a stack (' + ', ' ('), encountering a variable 3, adding a list (3,3), encountering an operator-, pushing a stack (' + ', '); encountering a variable 0, adding a list (3,3, 0).
A delimiter ") is encountered, and the operators in the stack are popped up and placed in the list in turn until a" ("is encountered. This operation pops up the operator ' - ' to add the list ('3', '3', '0', '); encountering the operator, ' + ', ') pushing stack; variable 2 is encountered and the list ('3', '3', '0', '-', '2') is added.
The data in the operator stack is sequentially pushed out and added to the list ('3', '3', '0', ','2',' + "), which is the final list of resolvable expressions expresstonkens.
2. As shown in fig. 2 as a flow chart of a feature expression parsing algorithm of a distributed network feature calculation method of the present invention,
the characteristic expression analysis algorithm comprises the following steps:
acquiring a generated solvable expression list expressTokens (3, 0, 2, + and plus);
when the traversal list expressTokens encounters a variable, the variable is pushed into a stack; the first three lists are all traversed, and the current stack is stack (3,3, 0);
when an operator is encountered, "-", the stack (3,3,0) pops a variable "3", "0", and a data stack (3) is left in the stack, the variable and the operator are calculated together to obtain a result 3-0 ═ 3, and the result "3" is pushed to the stack statck (3, 3);
encountering variable "2", pushing the variable to the stack (3,3,2)
When an operator is met, a variable "3" is popped from the stack (3,3,2), a data stack (3) is left in the stack, the variable and the operator calculate to obtain a result 3 x 2 ═ 6, and the result "6" is pushed to the stack statck (3, 6);
when the operator "+" is met, the stack (3,6) pops up the variable "3", "6", and the current stack has no data, the variable is calculated together with the operator to obtain the result 3+6 ═ 9, and finally the result 9 is returned.
3. As shown in figure 3 which is a flow chart of the distributed route calculation of the distributed network characteristic calculation method of the present invention,
the distributed routing computation includes:
after the networking of the social network is finished, the data of the network ID, the nodes and the edges are saved;
the nodes involved in the feature calculation and the weight values of the corresponding nodes are inquired and shown in FIG. 4;
distributing a social network to be calculated for the nodes according to a weight algorithm and storing the social network into redis, as shown in fig. 5;
broadcasting a notification to the node machines participating in the computation, as shown in fig. 6;
after receiving the 'calculation characteristic message', each node machine pulls the network distributed to itself to perform characteristic calculation.
The above-mentioned embodiments only express the specific embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for those skilled in the art, without departing from the technical idea of the present application, several changes and modifications can be made, which are all within the protection scope of the present application.
Claims (3)
1. A distributed network feature computation method is characterized by comprising the following steps: generating and analyzing a characteristic expression and calculating a distributed route;
the characteristic expression generation and analysis comprises the following steps: extracting elements in the characteristic expression to form an element queue; establishing a list structure storage unit and a stack structure storage unit; popping all the remaining elements in the stack structure storage unit out of the stack structure storage unit according to a last-in first-out sequence, and sequentially putting the elements into the list structure storage unit to form an analyzable list; analyzing the characteristic expression;
the distributed routing computation includes: after the networking of the social network is finished, the data of the network ID, the nodes and the edges are saved; inquiring nodes participating in feature calculation and weight values of corresponding nodes; and distributing the social network to be calculated for the nodes according to the weight algorithm and storing the social network to be calculated in a redis database, notifying the node machines participating in calculation by broadcasting, and pulling the social network distributed to each node for characteristic calculation after the node machines receive the message of calculating the characteristic task.
2. The method of claim 1, wherein the feature expression generation and parsing further comprises:
dividing elements in the feature expression into three types of variables, operators and separators, establishing a queue structure storage unit in a memory of the feature analysis device, and extracting and storing all elements in the feature expression into the queue structure storage unit one by one according to a left-to-right sequence to form an element queue;
establishing a list structure storage unit and a stack structure storage unit; traversing the element queue, sequentially putting variables in the element queue into a list structure storage unit, sequentially putting operators and separators into a stack structure storage unit, and sequentially popping the separators and the related operators from the stack structure storage unit and putting the separators and the related operators into the list structure storage unit according to the operation priority attribute of the separators put into the stack structure storage unit;
after traversing the element queue, popping all the remaining elements in the stack structure storage unit out of the stack structure storage unit according to a last-in first-out sequence, and sequentially putting the elements into the list structure storage unit, thereby forming an analyzable list;
traversing the analyzable list, sequentially putting elements in the analyzable list into a stack structure storage unit, if the put elements are operators, popping variables related to the operators in the stack structure storage unit from the stack structure storage unit respectively and operating the variables with the operators, and putting the operated result into the stack structure storage unit until no element exists in the stack structure storage unit.
3. The distributed network feature computation method of claim 1, wherein the distributed route computation further comprises:
each node machine participating in calculation is stored in a node detail table of the database, the node detail table comprises a weight value of each node machine, when a social network to be calculated is distributed, the weight value of each node machine is used as a weight proportion to carry out weighting calculation, and a calculated result is stored in a redis with a host name of key;
according to the published subscription message carried by the redis, each node machine participating in the calculation can subscribe a fixed message of 'calculation characteristic task', when the message is published, the node subscribing the message can receive the message data, and after the message is received, each calculation node machine pulls a corresponding standby calculation network from the redis through the host name key to perform characteristic calculation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011357397.1A CN112702264A (en) | 2020-11-27 | 2020-11-27 | Distributed network feature calculation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011357397.1A CN112702264A (en) | 2020-11-27 | 2020-11-27 | Distributed network feature calculation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112702264A true CN112702264A (en) | 2021-04-23 |
Family
ID=75506675
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011357397.1A Pending CN112702264A (en) | 2020-11-27 | 2020-11-27 | Distributed network feature calculation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112702264A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102404615A (en) * | 2011-11-29 | 2012-04-04 | 广东威创视讯科技股份有限公司 | Video processing system based on cloud computing |
| US20130205028A1 (en) * | 2012-02-07 | 2013-08-08 | Rackspace Us, Inc. | Elastic, Massively Parallel Processing Data Warehouse |
| CN110995725A (en) * | 2019-12-11 | 2020-04-10 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111008020A (en) * | 2019-12-24 | 2020-04-14 | 四川新网银行股份有限公司 | Method for analyzing logic expression into general query statement |
| CN111444236A (en) * | 2020-03-23 | 2020-07-24 | 华南理工大学 | Mobile terminal user portrait construction method and system based on big data |
| CN111787055A (en) * | 2020-05-22 | 2020-10-16 | 中国科学院信息工程研究所 | Redis-based transaction mechanism and multi-data center oriented data distribution method and system |
| CN111966719A (en) * | 2020-10-21 | 2020-11-20 | 四川新网银行股份有限公司 | Method for refreshing local data cache of distributed consumer credit system in real time |
-
2020
- 2020-11-27 CN CN202011357397.1A patent/CN112702264A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102404615A (en) * | 2011-11-29 | 2012-04-04 | 广东威创视讯科技股份有限公司 | Video processing system based on cloud computing |
| US20130205028A1 (en) * | 2012-02-07 | 2013-08-08 | Rackspace Us, Inc. | Elastic, Massively Parallel Processing Data Warehouse |
| CN110995725A (en) * | 2019-12-11 | 2020-04-10 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111008020A (en) * | 2019-12-24 | 2020-04-14 | 四川新网银行股份有限公司 | Method for analyzing logic expression into general query statement |
| CN111444236A (en) * | 2020-03-23 | 2020-07-24 | 华南理工大学 | Mobile terminal user portrait construction method and system based on big data |
| CN111787055A (en) * | 2020-05-22 | 2020-10-16 | 中国科学院信息工程研究所 | Redis-based transaction mechanism and multi-data center oriented data distribution method and system |
| CN111966719A (en) * | 2020-10-21 | 2020-11-20 | 四川新网银行股份有限公司 | Method for refreshing local data cache of distributed consumer credit system in real time |
Non-Patent Citations (1)
| Title |
|---|
| 李源昊等: ""面向移动社会网络的用户年龄与性别特征识别"", 《计算机应用》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105430100A (en) | User value assessment method and device, and application method and system thereof | |
| CN112566093B (en) | Terminal relation identification method and device, computer equipment and storage medium | |
| CN112800095A (en) | Data processing method, device, equipment and storage medium | |
| CN110032576A (en) | A kind of method for processing business and device | |
| CN103248677A (en) | Internet behavior analysis system and working method thereof | |
| Drezner et al. | A directional approach to gradual cover | |
| CN114282011A (en) | Knowledge graph construction method and device, and graph calculation method and device | |
| CN112365109B (en) | Distribution path information generation method and device, electronic equipment and readable medium | |
| CN110175438A (en) | Share account detection method and relevant device | |
| CN113569162A (en) | Data processing method, device, equipment and storage medium | |
| CN117291722A (en) | Object management method, related device and computer readable medium | |
| CN104834958B (en) | A kind of method and apparatus judged the step of answer | |
| CN112702264A (en) | Distributed network feature calculation method | |
| de Andrade Júnior et al. | Fully retroactive minimum spanning tree problem | |
| KR102457359B1 (en) | Marketing cost efficiency calculation method and electronic system for performing the method using neural networks | |
| Zhang et al. | Role of subgraphs in epidemics over finite-size networks under the scaled SIS process | |
| CN109697224B (en) | Bill message processing method, device and storage medium | |
| CN110457387A (en) | A kind of method and relevant apparatus determining applied to user tag in network | |
| CN107767278B (en) | Method and device for constructing community hierarchy | |
| CN111353001B (en) | Method and device for classifying users | |
| CN107171949B (en) | Information pushing method, device and system | |
| CN111966916A (en) | Recommendation method and device, electronic equipment and computer readable storage medium | |
| EP4318375A1 (en) | Graph data processing method and apparatus, computer device, storage medium and computer program product | |
| KR102457356B1 (en) | Marketing cost efficiency calculation method and electronic system for performing the method using neural networks | |
| Mukul et al. | Evaluation of smart cities with integrated hesitant fuzzy linguistic AHP–COPRAS method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210423 |