CN115018057A - Robust neural architecture searching method and system for graph neural network - Google Patents
Robust neural architecture searching method and system for graph neural network Download PDFInfo
- Publication number
- CN115018057A CN115018057A CN202210744891.6A CN202210744891A CN115018057A CN 115018057 A CN115018057 A CN 115018057A CN 202210744891 A CN202210744891 A CN 202210744891A CN 115018057 A CN115018057 A CN 115018057A
- Authority
- CN
- China
- Prior art keywords
- gnn
- search
- architecture
- searching
- framework
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 41
- 230000001537 neural effect Effects 0.000 title claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 45
- 239000011159 matrix material Substances 0.000 claims description 52
- 241000689227 Cora <basidiomycete fungus> Species 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims description 6
- 238000010200 validation analysis Methods 0.000 claims description 6
- 230000002708 enhancing effect Effects 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 4
- 235000009508 confectionery Nutrition 0.000 claims description 3
- 230000006870 function Effects 0.000 description 9
- 238000011156 evaluation Methods 0.000 description 8
- 238000005457 optimization Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 238000010845 search algorithm Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a robust neural architecture searching method and a system thereof for a graph neural network, wherein the searching method comprises the steps of defining a searching unit, a searching space and a GNN architecture; searching a plurality of GNN frameworks in a search space, and training all the GNN frameworks; adopting an anti-attack method to generate an anti-sample, inputting the anti-sample into each trained GNN framework, and taking the output accuracy as the robustness index; searching a search unit meeting preset conditions in the GNN framework with adjacent subscripts according to the robustness index of the GNN framework, and storing the search unit to a search unit set; according to the search unit set B * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy; constructing by adopting the operation corresponding to the maximum weight of each searched edgeForming a GNN neural network architecture.
Description
Technical Field
The invention relates to the technical field of neural networks, in particular to a robust neural architecture searching method and a robust neural architecture searching system for a graph neural network.
Background
In recent years, along with the development of NAS technology, the high efficiency and the design method for neural architecture automation are receiving more and more attention from GNN-related researchers. Currently, methods for designing GNN models based on NAS are mainly divided into three types: reinforcement learning based methods, evolutionary computation based methods, and gradient optimization based methods.
Although the above methods can all search for GNN models with high accuracy over a variety of map-related tasks, they are primarily directed to the accuracy of GNN models deployed in the absence of counter-attacks. However, in practical applications, due to the existence of a countermeasure sample (data generated after adding disturbance in input data by a specific algorithm, the disturbance is mainly the addition or deletion of edges and nodes in graph data), the robustness of the GNN model is also a problem to be considered.
The existing NAS method aiming at the GNN does not consider the robustness of the searched GNN model, so that the robustness of the searched GNN model is poor when confronted with challenge samples. For example, when the GNN is applied to identify a rogue user in a financial field-related system, the rogue user can modify the input data of the GNN by establishing contact with a user with high confidence, so that the GNN predicts it as a user with good confidence, which greatly reduces the security of the financial system.
Furthermore, when GNNs are applied to the identification and filtering of spam, etc., the manufacturer of spam, etc. can associate such mail, ads, etc. with normal mail, ads, causing the GNN filter to make incorrect predictions, which can lead to the failure of the spam, ad filtering system.
Therefore, a neural network architecture method considering the accuracy of the searched GNN architecture and the robustness against the attack needs to be proposed.
Disclosure of Invention
Aiming at the defects in the prior art, the robust neural architecture searching method and the system thereof for the graph neural network provided by the invention solve the problem of poor robustness of the existing method.
In order to achieve the purpose of the invention, the invention adopts the technical scheme that:
in a first aspect, a robust neural architecture search method for a graph neural network is provided, which includes:
s1, defining a search unit as one edge and a binary group formed by operations contained in the edge, wherein the search space S and the GNN framework are both a set formed by a plurality of search units;
s2, searching a plurality of GNN frameworks in the search space, and training all the GNN frameworks;
s3, generating a countermeasure sample by adopting an countermeasure attack method, inputting the countermeasure sample into each trained GNN framework, and taking the output accuracy as the robustness index;
s4, searching a search unit meeting preset conditions in the GNN framework with adjacent subscripts according to the robustness index of the GNN framework, and storing the search unit into a search unit set;
s5, collecting B according to search units * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy;
and S6, constructing and forming a GNN neural network architecture by adopting the operation corresponding to the searched maximum weight of each edge.
Further, the step S4 includes:
s41, searching a GNN framework with adjacent subscripts in the framework set; determining GNN architecture A i Whether the robustness index of (2) is greater than the GNN framework A i+1 The robustness index of (2);
s42, if yes, calculating A i -A i+1 The obtained partial search unit is stored in the set B 1 Performing the following steps;
s43, otherwise, calculating A i+1 -A i The obtained partial search unit is stored in the set B 2 Performing the following steps;
s44, set B is obtained 1 And B 2 And storing the search unit in the intersection of the search units into the search unit set B * In (1).
Further, the step S5 includes:
s51, reading the weight of each operation of each edge in the search space to form a weight matrix alpha, and constructing an initial matrix alpha which has the same dimension as the weight matrix alpha and has zero elements R ；
S52, initializing the matrix alpha R Corresponds to a search unit set B * The weight of the position of the middle search cell is set to 1, and an initial matrix alpha is calculated R A distance metric ψ (α) from the weight matrix α;
s53, obtaining a pair search unit set B according to the distance measurement * The weight matrix alpha after the probability enhancement is carried out by the medium search unit:
s.t.ω * (α)＝argmin ω L train (ω，α)
wherein L is train (omega, alpha) and L val (ω, α) is the loss of GNN framework on the training/validation set, respectively; λ is a weight variable; ω is a trainable parameter in the GNN architecture; omega * (α) is the loss L of the GNN framework on the training set train The value of the trainable parameter ω corresponding to the minimum;
s54, selecting the operation corresponding to the maximum weight of each edge in the search space according to the weight matrix alpha after probability enhancement:
wherein o is (i，j) An operation corresponding to the maximum weight of the edge; argmax is a function of the value of an independent variable when a function is solved and the maximum value is obtained;
a set of all operations for one edge;
is the weight of operation o on edge (i, j); i and j are the start and end points of the edge, respectively.
Further, the distance metric ψ (α) is calculated by the formula:
wherein | · | purple sweet 2 Is the euclidean distance.
Further, the number of search cells in the search space S is greater than the number of search cells in the GNN architecture.
Further, when the constructed GNN neural network architecture is used for classifying scientific documents, the data set for training the GNN architecture is a Cora data set and a Citeseer data set;
when the constructed GNN neural network architecture is used for classifying junk mails in a mail system, a data set for training the GNN architecture is a Spambase data set;
when the constructed GNN neural network architecture is used for identifying a cheater user in a financial system, a data set for training the GNN architecture is a PaySim data set;
when the constructed GNN neural network architecture is used for adding a large amount of fake user scoring information into a recommendation system and defending malicious users trying to change recommendation results, a data set for training the GNN architecture is a Retailocket data set.
In a second aspect, a robust neural architecture search system for a graph-oriented neural network is provided, which includes:
the definition module is used for defining the search units as a side and a binary group formed by operations contained in the side, and the search space S and the GNN framework are both a set formed by a plurality of search units;
the GNN architecture search training module is used for searching a plurality of GNN architectures in a search space and training all the GNN architectures;
the robustness index generation module is used for generating countermeasure samples by adopting an countermeasure attack method, inputting the countermeasure samples into each trained GNN framework, and taking the output accuracy as the robustness index;
the search module is used for searching a search unit which meets preset conditions in the GNN framework with adjacent subscripts according to the robustness index of the GNN framework, and storing the search unit to a search unit set;
a probability enhancing module for collecting B according to the search units * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy;
and the network architecture generating module is used for constructing and forming the GNN neural network architecture by adopting the operation corresponding to the searched maximum weight of each edge.
Further, the search module includes:
the searching and judging module is used for searching the GNN framework with adjacent subscripts in the framework set; determining GNN architecture A i Whether the robustness index of (2) is greater than the GNN framework A i+1 The robustness index of (2);
a first search unit calculation module for calculating A when the output of the search judgment module is yes i -A i+1 The obtained partial search unit is stored in the set B 1 Performing the following steps;
a second search unit calculating module for calculating A when the output of the judging module is not found i+1 -A i The obtained partial search unit is stored in the set B 2 Performing the following steps;
a search unit determination module for finding the set B 1 And B 2 And storing to search unit set B * In (1).
Further, the probability enhancing module comprises:
a matrix construction module for reading the weight of each operation of each edge in the search space to form a weight matrix alpha, and constructing a matrix with the same dimension as the weight matrix alpha and zero elementsInitial matrix alpha of R ；
A distance metric calculation module for calculating an initial matrix alpha R Corresponds to a search unit set B * The weight of the position of the middle search cell is set to 1, and an initial matrix alpha is calculated R A distance metric ψ (α) from the weight matrix α;
a probability enhancement submodule for obtaining a search unit set B according to the distance measurement * The weight matrix alpha after the probability enhancement is carried out by the medium search unit:
s.t.ω * (α)＝argmin ω L train (ω，α)
wherein L is train (omega, alpha) and L val (ω, α) is the loss of GNN framework on the training/validation set, respectively; λ is a weight variable; ω is a trainable parameter in the GNN architecture; omega * (α) is the loss L of the GNN framework on the training set train The value of the trainable parameter ω corresponding to the minimum;
an operation selection module, configured to select, according to the weight matrix α after probability enhancement, an operation corresponding to the maximum weight of each edge in the search space:
wherein o is (i，j) An operation corresponding to the maximum weight of the edge; argmax is a function of the value of an independent variable when a function is solved and the maximum value is obtained;
a set of all operations for one edge;
is the weight of operation o on edge (i, j); i and j are the start and end points of the edge, respectively.
The invention has the beneficial effects that: according to the scheme, from the perspective of a search space, the robust search unit can be marked in the search space. Based on the marked robust search units, in the search process, the probability enhancement is carried out on the robust search units in the search space while the framework precision is ensured, so that as many robust search units as possible are selected into the final framework, and the robustness of the final GNN neural network framework is improved.
Drawings
Fig. 1 is a flowchart of a robust neural architecture search method for a graph neural network.
FIG. 2 is a search unit set B * And searching the acquisition idea diagram of the unit.
Fig. 3 is a comparison of the performance of GNN architectures as the attack strength increases.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.
Referring to fig. 1, fig. 1 shows a flow diagram of a robust neural architecture search method for a graph-oriented neural network; as shown in fig. 1, the method S includes steps S1 to S6.
In step S1, a search cell is defined as a binary group formed by operations contained in one edge and one edge, the search space S and the GNN structure are both a set formed by a plurality of search cells, and the number of search cells in the search space S is greater than the number of search cells in the GNN structure. The specific implementation process of step S1 is as follows:
in the scheme, a search unit is defined as a side e and a binary group formed by operation o contained in the side, and is marked as (e, o); based on the above definition, the search space S can be defined as a set of search units, denoted as
Where ε represents the set of all edges in the search space,
it represents the set of all operations that may be contained on one edge. Similarly, similar to the definition of the search space S, a GNN structure is also defined as a set of search units, and a structure containing n search units can be denoted as { b 1 ，b 2 ，...，b n |，b i ∈S，i＝1，2，...，n}。
In step S2, a number of GNN architectures are searched in the search space, and all GNN architectures are trained; according to the scheme, a GNN architecture search is carried out by adopting a running search algorithm, and 50 GNN architectures can be obtained by assuming that 50 periods are set (one GNN architecture can be obtained in each period).
During implementation, when the scheme preferentially trains the GNN framework, the data set is selected in the following mode:
when the constructed GNN neural network architecture is used for classifying scientific documents, a data set for training the GNN architecture is a Cora data set and a Citeseer data set;
when the constructed GNN neural network architecture is used for classifying junk mails in a mail system, a data set for training the GNN architecture is a Spambase data set;
when the constructed GNN neural network architecture is used for identifying a cheater user in a financial system, a data set for training the GNN architecture is a PaySim data set;
when the constructed GNN neural network architecture is used for adding a large amount of fake user scoring information into a recommendation system and defending malicious users trying to change recommendation results, a data set for training the GNN architecture is a Retailocket data set.
In step S3, a counterattack method is used to generate countersamples, the countersamples are input into each trained GNN architecture, and the accuracy of the output is used as its robustness index.
In step S4, search units satisfying a preset condition in the GNN architecture with adjacent subscripts are searched according to the robustness index of the GNN architecture, and stored in the search unit set.
In one embodiment of the present invention, the step S4 includes:
s41, searching a GNN framework with adjacent subscripts in the framework set; determining GNN architecture A i Whether the robustness index of (A) is greater than that of the GNN architecture A i+1 The robustness index of (2);
s42, if yes, calculating A i -A i+1 The obtained partial search unit is stored in the set B 1 Performing the following steps;
s43, otherwise, calculating A i+1 -A i The obtained partial search unit is stored in the set B 2 Performing the following steps;
s44, set B is obtained 1 And B 2 And storing to search unit set B * In (1).
According to the scheme, the search unit with better robustness is obtained by adopting the mode and is used for constructing a robust search space, and the robust GNN framework is more favorably searched on the basis of the search space.
To facilitate the collection of search units B * The following describes a specific implementation process of step S4 with reference to fig. 2, in order to understand the acquisition process of the medium search unit:
for architecture A in FIG. 2 m And A m+1 Assuming that their robustness satisfies the relation R m ＞R m+1 Then A should be selected m -A m+1 The search unit in (A) is added to the set B 2 In (1). Framework A m And A m+1 There is a different search unit, indicated by the dashed arrow in the figure, then (0, 3, operation 3) this search unit should be added to the set B 2 Among them.
Similarly, for architecture A n And A n+1 Their robustness satisfies R n ＜R n+1 Therefore, A should be selected n+1 -A n The search unit in (A) is added to the set B 1 Among them. In the framework A n And A n+1 There is also a different search unit, indicated by the dashed arrow in the figure. Then, A n+1 -A n The search unit included in (1) is ((0, 3), operation 3), and this search unit should be added to B 1 In (1).
Finally, when set B 1 And set B 2 When the intersection is taken ((0, 3), operation 3) the search unit is stored in the search unit set B * In (1).
In step S5, set B of search units * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy;
in one embodiment of the present invention, the step S5 includes:
s51, reading the weight of each operation of each edge in the search space to form a weight matrix alpha, and constructing an initial matrix alpha which has the same dimension as the weight matrix alpha and has zero elements R ；
S52, initializing the matrix alpha R Corresponds to a search unit set B * The weight of the position of the middle search cell is set to 1, and an initial matrix alpha is calculated R Distance metric ψ (α) from weight matrix α:
wherein | · | purple sweet 2 Is the euclidean distance.
S53, obtaining a pair search unit set B according to the distance measurement * The weight matrix alpha after the probability enhancement is carried out by the medium search unit:
s.t.ω * (α)＝argmin ω L train (ω，α)
wherein L is train (omega, alpha) and L val (ω，α) is the loss of GNN framework on the training/validation set, respectively; λ is a weight variable; ω is a trainable parameter in the GNN architecture; omega * (α) is the loss L of the GNN framework on the training set train The value of the trainable parameter ω corresponding to the minimum;
s54, selecting the operation corresponding to the maximum weight of each edge in the search space according to the weight matrix alpha after probability enhancement:
wherein o is (i，j) An operation corresponding to the maximum weight of the edge; argmax is a function of the value of an independent variable when a function is solved and the maximum value is obtained;
a set of all operations for one edge;
is the weight of operation o on edge (i, j); i and j are the start and end points of the edge, respectively.
In step S6, the GNN neural network architecture is constructed and formed by using the operation corresponding to the searched maximum weight of each edge.
The scheme also provides a robust neural architecture search system for the graph neural network, which comprises the following steps:
the definition module is used for defining the search units as a side and a binary group formed by operations contained in the side, and the search space S and the GNN framework are both a set formed by a plurality of search units;
the GNN architecture search training module is used for searching a plurality of GNN architectures in a search space and training all the GNN architectures;
the robustness index generation module is used for generating countermeasure samples by adopting an countermeasure attack method, inputting the countermeasure samples into each trained GNN framework, and taking the output accuracy as the robustness index;
the search module is used for searching a search unit which meets preset conditions in the GNN framework with adjacent subscripts according to the robustness index of the GNN framework, and storing the search unit to a search unit set;
in implementation, the preferred search module in the scheme comprises:
the searching and judging module is used for searching the GNN framework with adjacent subscripts in the framework set; determining GNN framework A i Whether the robustness index of (A) is greater than that of the GNN architecture A i+1 The robustness index of (2);
a first search unit calculation module for calculating A when the output of the search judgment module is yes i -A i+1 The obtained partial search unit is stored in the set B 1 Performing the following steps;
a second searching unit calculating module for calculating A when the output of the judging module is not searched i+1 -A i The obtained partial search unit is stored in the set B 2 Performing the following steps;
a search unit determination module for finding the set B 1 And B 2 And storing to search unit set B * In (1).
A probability enhancing module for collecting B according to the search units * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy;
in one embodiment of the present invention, the probability enhancing module comprises:
a matrix construction module for reading the weight of each operation of each edge in the search space to form a weight matrix alpha and constructing an initial matrix alpha which has the same dimensionality as the weight matrix alpha and has zero elements R ；
A distance metric calculation module for calculating an initial matrix alpha R Corresponds to a search unit set B * The weight of the position of the middle search cell is set to 1, and an initial matrix alpha is calculated R A distance metric ψ (α) from the weight matrix α;
a probability enhancement submodule for obtaining a search unit set B according to the distance measurement * The weight matrix alpha after the probability enhancement is carried out by the medium search unit:
s.t.ω * (α)＝argmin ω L train (ω，α)
wherein L is train (omega, alpha) and L val (ω, α) is the loss of GNN framework on the training/validation set, respectively; λ is a weight variable; ω is a trainable parameter in the GNN architecture; omega * (α) is the loss L of the GNN framework on the training set train The value of the trainable parameter ω corresponding to the minimum;
an operation selection module, configured to select, according to the weight matrix α after probability enhancement, an operation corresponding to the maximum weight of each edge in the search space:
wherein o is (i，j) An operation corresponding to the maximum weight of the edge; argmax is a function of the value of an independent variable when a function is solved and the maximum value is obtained;
a set of all operations for one edge;
is the weight of operation o on edge (i, j); i and j are the start and end points of the edge, respectively.
And the network architecture generating module is used for constructing and forming the GNN neural network architecture by adopting the operation corresponding to the searched maximum weight of each edge.
The performance evaluation is performed on the GNN neural network architecture searched by the scheme by combining the specific cases as follows:
A. data set construction and evaluation index
The performance of the GNN neural network architecture constructed by the present solution was evaluated using the GNN standard dataset Cora and CiteSeer, where the evaluation is divided into two parts: 1) inputting the original data in the data set into the GNN, obtaining the output accuracy, and mainly evaluating the precision of the GNN architecture; 2) and inputting the data after disturbance is added to the original data of the data set by the anti-attack algorithm into the GNN, obtaining the output accuracy and mainly evaluating the robustness of the GNN architecture. The Cora and CiteSeer data sets contain multiple categories of scientific publications and their citations. Each node in the data set represents a scientific publication, and each edge represents that a citation relationship exists between two scientific publications.
And (3) evaluating the performance of the GNN framework under two conditions of countermeasure training and standard training by adopting the classification accuracy of the node classification task as an evaluation index of the performance of the GNN framework. The method comprises the steps of performing countermeasure training, namely generating countermeasure samples by adopting an countermeasure attack algorithm, taking the countermeasure samples as training data of a model, and mainly evaluating the robustness of the GNN framework when defensive measures exist. The standard training is training using raw data in the dataset, and the main evaluation is the robustness of the GNN architecture itself. In addition, to demonstrate the migratability of the schema, the schema searched on the Cora dataset was trained and evaluated directly on the CiteSeer dataset.
When the robustness of the architecture is evaluated, the scheme adopts three attack methods: random attack (random attach), Delete Internal Connect External (DICE) and Node embedding attack (Node embedding attach), all of which have a disturbance rate of 5% of the number of edges in the graph.
B. Details of the experiment
When the performance of the method is evaluated, the method is applied to a commonly used NAS (network attached storage) method SANE (generalized adaptive network access network) based on gradient in GNN (GNN), the robust GNN architecture is searched, and the performance of the searched GNN architecture is evaluated.
C. Algorithm parameter setting
And (3) searching: following the general setup of a search algorithm based on Gradient optimization, Epoch is set to 200, and the learnable parameters of the optimized architecture are attenuated using a random Gradient Descent (SGD) with a learning rate of 0.025, momentum of 0.9 and weight of 0.0003. Adam was used to optimize the architecture parameters at a learning rate of 0.0003, weight decay of 0.001. The weight parameter λ in the objective function is set to 0.01.
And (3) evaluation process: the model was finally evaluated for performance after 600 epochs of model training using SGD with a learning rate of 0.025, a momentum of 0.9 and a weight decay of 0.0003. To achieve a fair comparison, all GNN models were trained using the same settings.
D. Comparison of results
The method comprises the following steps of (REP), the comparison method comprises a manually constructed GNN model (GCN, GAT and GraphSAGE), and a model (SANE) searched based on a gradient optimization method. All data are mean and standard deviation values determined from node classification accuracy (%) obtained from five training sessions of model records. The best results are marked in bold.
Table 1 evaluation results of the confrontational training under the Cora data set
As can be seen from the results in table 1, when all GNN models are subjected to countermeasure training, the classification accuracy of the GNN model searched by the method is higher than that of other GNN models in both the presence and absence of the countermeasure attack. The method has the advantages that the GNN model searched by the method has better accuracy and robustness in case of defensive measures than the manually constructed GNN model and the GNN model searched by the gradient optimization method.
Table 2 evaluation results of standard training under the Cora data set
From the results in table 2, when all models are subjected to standard training, the model searched by the method according to the present invention still has higher accuracy than other GNN models under both the original data and the attack resistance. When the influence of the countertraining on the robustness of the models is removed, the difference in robustness among the models mainly comes from the internal structure of the models. Therefore, the experimental data in table 2 prove that the GNN model searched by the method of the present invention has better robustness than other GNN models in terms of model structure.
TABLE 3 evaluation results of migratability under CiteSeer dataset
From the results in table 3, it can be seen that when the GNN model searched on the Cora data set by the method according to the present embodiment is directly trained on the CiteSeer data set, the GNN model according to the present embodiment achieves the highest classification accuracy in both the classification accuracy of the original data and the classification accuracy in the presence of counterattack. In particular, in the experiments, the present solution keeps each GNN model the same as it was when experimented on the Cora dataset to ensure fairness of the comparison. Experimental results show that the GNN model searched by the method has good mobility, and good classification accuracy and robustness can be kept on other data sets.
Meanwhile, in order to more fully show the superior robustness of the REP in the scheme, the disturbance rate of the DICE anti-attack algorithm is set to be 0%, 10%, 20%, 30%, 40% and 50% in sequence, so as to compare the performances of the GNN architectures when the attack strength is gradually increased, and the comparison result is shown in FIG. 3.
From the results of fig. 3, it can be easily found that the performance degradation of GCN, GAT, GraphSAGE and SANE is very significant and large in the case of increasing strength against attack. Compared with the GNN framework searched by the REP method, the descending rate of the GNN framework searched by the REP method is small, the highest accuracy rate in all models is always kept, and the GNN framework searched by the REP method has good robustness.
E. Advantages in practical scenarios
Similar to the node classification task on the Cora and CiteSeer datasets, when GNN is applied to an actual scene, such as: the problems to be solved in the detection of cheater users, the detection and filtration of junk mails and junk advertisements in the financial system are still the node classification problem. Specifically, in the detection of rogue users in financial systems, the task of GNN is to classify each node in the input data as a rogue user or a user with good credit, and in the detection and filtering of spam and spam, the task of GNN is to classify the nodes in the input data as normal mail, advertisement or spam, advertisement.
Compared with the traditional manually constructed GNN models such as GCN, GAT and GraphSAGE and the GNN models searched by the NAS method based on gradient optimization such as SANE, the GNN models with good classification accuracy and robustness in the node classification task can be searched by the method. Thus, after the map data input to the GNN is perturbed, as: the cheat user in the input data is added with the user with good credit, the junk mail, the advertisement is added with the normal mail and the advertisement, and the like.
In summary, since the GNN architecture searched by the method of the present invention has good robustness, when the method is applied to an actual scene, the method of the present invention is very advantageous for improving the security of the system in the actual application scene.
Claims (9)
1. The robust neural architecture searching method for the graph neural network is characterized by comprising the following steps of:
s1, defining a search unit as one edge and a binary group formed by operations contained in the edge, wherein the search space S and the GNN framework are both a set formed by a plurality of search units;
s2, searching a plurality of GNN frameworks in the search space, and training all the GNN frameworks;
s3, generating a countermeasure sample by adopting an countermeasure attack method, inputting the countermeasure sample into each trained GNN framework, and taking the output accuracy as the robustness index;
s4, searching a search unit meeting preset conditions in the GNN framework with adjacent subscripts according to the robustness index of the GNN framework, and storing the search unit into a search unit set;
s5, collecting B according to search units * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy;
and S6, constructing and forming a GNN neural network architecture by adopting the operation corresponding to the searched maximum weight of each edge.
2. The robust neural architecture search method for neural network-oriented according to claim 1, wherein the step S4 includes:
s41, searching a GNN framework with adjacent subscripts in the framework set; determining GNN architecture A i Whether the robustness index of (A) is greater than that of the GNN architecture A i+1 The robustness index of (2);
s42, if yes, calculating A i -A i+1 The obtained partial search unit is stored in the set B 1 Performing the following steps;
s43, otherwise, calculating A i+1 -A i The obtained partial search unit is stored in the set B 2 Performing the following steps;
s44, set B is obtained 1 And B 2 And storing the search unit in the intersection of the search units into the search unit set B * In (1).
3. The robust neural architecture search method for neural network-oriented according to claim 1, wherein the step S5 includes:
s51, reading the weight of each operation of each edge in the search space to form a weight matrix alpha, and constructing an initial matrix alpha which has the same dimension as the weight matrix alpha and has zero elements R ；
S52, initializing the matrix alpha R Corresponds to a search unit set B * The weight of the position of the middle search cell is set to 1, and an initial matrix alpha is calculated R With a weight matrix alphaA distance metric ψ (α);
s53, obtaining a pair search unit set B according to the distance measurement * The weight matrix alpha after the probability enhancement is carried out by the medium search unit:
s.t.ω * (α)＝argmin ω L train (ω，α)
wherein L is train (omega, alpha) and L val (ω, α) is the loss of GNN framework on the training/validation set, respectively; λ is a weight variable; ω is a trainable parameter in the GNN architecture; omega * (α) is the loss L of the GNN framework on the training set train The value of the trainable parameter ω corresponding to the minimum;
s54, selecting the operation corresponding to the maximum weight of each edge in the search space according to the weight matrix alpha after probability enhancement:
wherein, 0 (i，j) An operation corresponding to the maximum weight of the edge; argmax is a function of the value of its argument when a function is maximized,
a set of all operations for one edge;
is the weight of operation o on edge (i, j); i and j are the start and end points of the edge, respectively.
4. The robust neural architecture search method for graph-oriented neural networks according to claim 3, wherein the distance metric ψ (α) is calculated by the formula:
wherein | · | purple sweet 2 Is the euclidean distance.
5. The robust neural architecture search method for the graph-oriented neural network of claim 1, wherein the number of search units in the search space S is greater than the number of search units in the GNN architecture.
6. The robust neural architecture search method for graph-oriented neural networks as claimed in claim 1, wherein when the constructed GNN neural network architecture is used for classifying scientific literature, the data sets for training the GNN architecture are the Cora data set and the citeser data set;
when the constructed GNN neural network architecture is used for classifying junk mails in a mail system, a data set for training the GNN architecture is a Spambase data set;
when the constructed GNN neural network architecture is used for identifying a cheater user in a financial system, a data set for training the GNN architecture is a PaySim data set;
when the constructed GNN neural network architecture is used for adding a large amount of fake user scoring information into a recommendation system and defending malicious users trying to change recommendation results, a data set for training the GNN architecture is a Retailocket data set.
7. The robust neural architecture search system for the graph neural network is characterized by comprising the following components:
the definition module is used for defining the search units as a side and a binary group formed by operations contained in the side, and the search space S and the GNN framework are both a set formed by a plurality of search units;
the GNN architecture search training module is used for searching a plurality of GNN architectures in a search space and training all the GNN architectures;
the robustness index generation module is used for generating countermeasure samples by adopting an countermeasure attack method, inputting the countermeasure samples into each trained GNN framework, and taking the output accuracy as the robustness index;
the search module is used for searching a search unit which meets preset conditions in the GNN framework with adjacent subscripts according to the robustness index of the GNN framework, and storing the search unit to a search unit set;
a probability enhancing module for collecting B according to the search units * The searching unit searches the operation corresponding to the maximum weight of each edge in the searching space by adopting a probability enhancement searching strategy;
and the network architecture generating module is used for constructing and forming the GNN neural network architecture by adopting the operation corresponding to the searched maximum weight of each edge.
8. The robust neural architecture search system for neural networks according to claim 7, wherein said search module comprises:
the searching and judging module is used for searching the GNN framework with adjacent subscripts in the framework set; determining GNN architecture A i Whether the robustness index of (A) is greater than that of the GNN architecture A i+1 The robustness index of (2);
a first search unit calculation module for calculating A when the output of the search judgment module is yes i -A i+1 The obtained partial search unit is stored in the set B 1 Performing the following steps;
a second searching unit calculating module for calculating A when the output of the judging module is not searched i+1 -A i The obtained partial search unit is stored in the set B 2 Performing the following steps;
a search unit determination module for finding the set B 1 And B 2 And storing to search unit set B * In (1).
9. The robust neural architecture search system for neural networks according to claim 7, wherein the probability enhancement module comprises:
matrix construction module for read searchThe weight of each operation of each edge in the space forms a weight matrix alpha, and an initial matrix alpha which has the same dimension as the weight matrix alpha and has zero elements is constructed R ；
A distance metric calculation module for calculating an initial matrix alpha R Corresponds to a search unit set B * The weight of the position of the middle search cell is set to 1, and an initial matrix alpha is calculated R A distance metric ψ (α) from the weight matrix α;
a probability enhancement submodule for obtaining a search unit set B according to the distance measurement * The weight matrix alpha after the probability enhancement is carried out by the medium search unit:
s.t.ω * (α)＝argmin ω L train (ω，α)
wherein L is train (omega, alpha) and L val (ω, α) is the loss of GNN framework on the training/validation set, respectively; λ is a weight variable; ω is a trainable parameter in the GNN architecture; omega * (α) is the loss L of the GNN framework on the training set train The value of the trainable parameter ω corresponding to the minimum;
an operation selection module, configured to select, according to the weight matrix α after probability enhancement, an operation corresponding to the maximum weight of each edge in the search space:
wherein o is (i，j) An operation corresponding to the maximum weight of the edge; argmax is a function of the value of an independent variable when a function is solved and the maximum value is obtained;
a set of all operations for one edge;
is the weight of operation o on edge (i, j); i and j are the start and end points of the edge, respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210744891.6A CN115018057A (en) | 2022-06-28 | 2022-06-28 | Robust neural architecture searching method and system for graph neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210744891.6A CN115018057A (en) | 2022-06-28 | 2022-06-28 | Robust neural architecture searching method and system for graph neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115018057A true CN115018057A (en) | 2022-09-06 |
Family
ID=83077597
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210744891.6A Pending CN115018057A (en) | 2022-06-28 | 2022-06-28 | Robust neural architecture searching method and system for graph neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115018057A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115329088A (en) * | 2022-10-12 | 2022-11-11 | 中国人民解放军国防科技大学 | Robustness analysis method of graph neural network event detection model |
| CN117648946A (en) * | 2024-01-30 | 2024-03-05 | 南湖实验室 | DNN model automatic generation method for safety key system |
| CN117876221A (en) * | 2024-03-12 | 2024-04-12 | 大连理工大学 | Robust image splicing method based on neural network structure search |
-
2022
- 2022-06-28 CN CN202210744891.6A patent/CN115018057A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115329088A (en) * | 2022-10-12 | 2022-11-11 | 中国人民解放军国防科技大学 | Robustness analysis method of graph neural network event detection model |
| CN117648946A (en) * | 2024-01-30 | 2024-03-05 | 南湖实验室 | DNN model automatic generation method for safety key system |
| CN117876221A (en) * | 2024-03-12 | 2024-04-12 | 大连理工大学 | Robust image splicing method based on neural network structure search |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115018057A (en) | Robust neural architecture searching method and system for graph neural network | |
| CN108846422B (en) | Account number association method and system across social networks | |
| US11330009B2 (en) | Systems and methods for machine learning-based digital content clustering, digital content threat detection, and digital content threat remediation in machine learning task-oriented digital threat mitigation platform | |
| US8949158B2 (en) | Cost-sensitive alternating decision trees for record linkage | |
| Li et al. | Hierarchical rough decision theoretic framework for text classification | |
| Baek et al. | Bankruptcy prediction for credit risk using an auto-associative neural network in Korean firms | |
| CN112287997A (en) | Depth map convolution model defense method based on generative confrontation network | |
| CN112417176B (en) | Method, equipment and medium for mining implicit association relation between enterprises based on graph characteristics | |
| CN113407728B (en) | Knowledge graph construction and query recommendation system in radio signal attack and defense field | |
| CN117201122A (en) | Unsupervised attribute network anomaly detection method and system based on view level graph comparison learning | |
| CN116467666A (en) | Graph anomaly detection method and system based on integrated learning and active learning | |
| CN113159976B (en) | Identification method for important users of microblog network | |
| CN115936104A (en) | Method and apparatus for training machine learning models | |
| CN114722920A (en) | Deep map convolution model phishing account identification method based on map classification | |
| CN114494753A (en) | Clustering method, clustering device, electronic equipment and computer-readable storage medium | |
| Zhao | An evolutionary intelligent data analysis in promoting smart community | |
| Verma et al. | Detailed analysis of intrusion detection using machine learning algorithms | |
| CN117171639A (en) | Deep clustering-based efficient attack resistance method for graph neural network | |
| Zhao et al. | An Attention-based Long Short-Term Memory Framework for Detection of Bitcoin Scams | |
| Rath | Adaptive Modelling Approach for Row-Type Dependent Predictive Analysis (RTDPA): A Framework for Designing Machine Learning Models for Credit Risk Analysis in Banking Sector | |
| CN116596539B (en) | Money backwashing method and system | |
| Musa et al. | Investigation the Effect of Dataset Size on the Performance of Different algorithm in Phishing Website Detection | |
| Mahapatra et al. | Leveraging Multiple Online Sources for Accurate Income Verification | |
| CN117829634A (en) | Information policy processing method and terminal | |
| Yaman Kanmaz | Classification of Imbalanced Credit Data Sets with Borrower-Specific Cost-Sensitive Algorithms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220906 |