CN115905546A - Graph convolution network document identification device and method based on resistive random access memory - Google Patents
Graph convolution network document identification device and method based on resistive random access memory Download PDFInfo
- Publication number
- CN115905546A CN115905546A CN202310017218.7A CN202310017218A CN115905546A CN 115905546 A CN115905546 A CN 115905546A CN 202310017218 A CN202310017218 A CN 202310017218A CN 115905546 A CN115905546 A CN 115905546A
- Authority
- CN
- China
- Prior art keywords
- layer
- quantization
- graph convolution
- graph
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a graph convolution network document identification device and method based on a resistive random access memory, which are used for constructing a training set and a test set of a document identification data set; constructing a floating point graph convolution network model based on a resistive random access memory, and pre-training by using a training set to obtain pre-trained model parameters; constructing a graph convolution network quantification model based on a training phase of the resistive random access memory according to the floating point graph convolution network model; inputting the training set into a graph convolution network quantization model in a training stage, and performing quantization perception training to obtain the truncation bit width of each layer of output values, the weight of a loss function and model parameters after quantization perception training; constructing a graph convolution network quantitative model based on an inference phase of the resistive random access memory according to the graph convolution network quantitative model in the training phase; and mapping the model parameters after the quantitative sensing training to the resistive random access memory, and inputting the test set to a graph volume network quantitative model based on an inference stage of the resistive random access memory to perform forward inference test.
Description
Technical Field
The invention relates to the technical field of neural network literature identification, in particular to a graph volume network literature identification device and method based on a resistive random access memory.
Background
With the rapid development of deep learning, the graph convolution network technology has been widely applied to various fields such as text classification, recommendation systems, knowledge graph spectrum completion and the like. These applications typically need to be deployed at the edge device side. In a traditional chip architecture, a memory and a calculation are separated, a calculation unit reads data from the memory first, and the data is stored back to the memory after the calculation is completed. However, in the face of the high concurrency requirement of the neural network, the conventional chip architecture needs to frequently carry data, which results in huge power consumption and computation bottleneck.
The resistive random access memory has the advantages of low power consumption, simple structure, high working speed, controllable and variable resistance value and the like, and can realize various operation forms such as logic operation, matrix multiplication and the like. The characteristic of the resistive random access memory integrating storage can reduce data transportation and reduce storage requirements. Therefore, the resistive random access memory has great potential to solve the problems brought by the traditional chip architecture. In recent years, a graph convolution network accelerator based on a resistive random access memory provides an effective solution for reasoning of a graph convolution network.
Although the resistive random access memory has great advantages for realizing the inference of the graph convolution network, the graph convolution network model needs to be compressed in the implementation process, which causes the loss of precision. The reasonable and effective quantization method can reduce the storage space of data and improve the calculation speed under the condition of low precision loss. Because the conductance range of the resistive random access memory device is limited, a limited bit width is required to store the weight of the graph convolution network. Meanwhile, since the quantization bit width of the output value of the current layer is exceeded after the sub-operation of the graph volume calculation is performed, the output value of the current layer needs to be stored by using the output quantization bit width through truncation operation after the sub-operation of the graph volume calculation is performed. If the truncation bit width of the convolution output value of each layer of the graph is not optimized, the identification precision of the graph convolution network is reduced. Therefore, designing a reasonable and effective quantization method for specific hardware constraints is a difficult problem that researchers in the field need to overcome.
Disclosure of Invention
In order to solve the defects of the prior art, realize the consistency of the truncation bit width and the measuring range of an analog-to-digital converter and reduce the precision loss caused by quantization, the invention adopts the following technical scheme:
a graph convolution network document identification method based on a resistive random access memory comprises the following steps:
step S1: constructing a training set and a test set for a document identification data set;
step S2: constructing a floating point graph convolution network model based on a resistive random access memory, and performing model pre-training by using a training set to obtain pre-trained model parameters;
and step S3: constructing a graph convolution network quantification model based on a training phase of a resistive random access memory according to the floating point graph convolution network model;
and step S4: inputting the training set into a graph convolution network quantization model in a training stage, and performing quantization perception training to obtain a truncation bit width of each layer of output values, a weight of a loss function and model parameters after quantization perception training;
step S5: constructing a graph convolution network quantitative model based on an inference phase of the resistive random access memory according to the graph convolution network quantitative model in the training phase;
step S6: and mapping the model parameters after the quantitative perception training to the resistive random access memory, inputting the test set to a graph convolution network quantitative model based on an inference stage of the resistive random access memory, and performing forward inference test.
Further, the step S4 includes the steps of:
step S4-1: aggregating information of neighbor nodes through a feature aggregation layer for each document node of the training set;
step S4-2: performing quantization operation on the characteristic values of all the training set nodes obtained in the step S4-1 in the characteristic aggregation layer through an activation quantization layer to obtain quantized activation values;
step S4-3: the image convolution kernel is subjected to quantization operation through an image convolution quantization layer to obtain a quantized image convolution kernel, and the image convolution kernel is subjected to inverse quantization operation through an image convolution inverse quantization layer to obtain an inverse quantized image convolution kernel;
step S4-4: carrying out graph convolution operation on the inversely quantized activation value and the inversely quantized graph convolution kernel to obtain an inversely quantized graph convolution output value; obtaining the truncation bit width of the graph convolution output value based on the quantization factor and the quantization value, and in the same way, inversely quantizing the last graph convolution and then obtaining the output of the graph convolution network quantization model in the training stage through a classifier;
step S4-5: and updating network parameters including quantization factors of each layer, truncation bit width of convolution output values of each layer of the graph and weight of the loss function by optimizing the loss function until the network converges to obtain a graph convolution network quantization model after quantization perception training.
Further, the polymerization manner in step S4-1 is: acquiring the characteristics of a fixed number of first-order neighbor nodes of the node in a random sampling mode, cascading the characteristics with the characteristics of the node, then calculating the mean value of the cascaded characteristics in each characteristic dimension to be used as a new characteristic of the node, and obtaining the characteristic values of all training set nodes in a characteristic aggregation layer; the aggregation function is as follows:
wherein the content of the first and second substances,
representing a target node
The input characteristic value of (2) is,
first order neighbor node representing target node
The input characteristic value of (a) is,
meaning that the target node is concatenated with the input eigenvalues of all first order neighbor nodes,
representing a target node
The characteristic value of the characteristic aggregation layer.
Further, the activation value after quantization in the step S4-2:
then, carrying out inverse quantization operation by activating an inverse quantization layer, and obtaining an activation value after inverse quantization by a ReLU activation function:
wherein
Representing the floating-point eigenvalues of all training set nodes at the feature aggregation level,
it is meant to round-off the process,
it is indicated that the operation of truncation is performed,
the minimum value after the quantization is represented by,
the maximum value after the quantization is represented,
a quantization factor representing a floating point value of a training set node feature at a feature aggregation layer.
The graph convolution kernel after quantization in the step S4-3:
the graph convolution kernel after the dequantization:
wherein
Representing the graph convolution kernel floating point values,
it is meant to round-off the process,
it is indicated that the operation of truncation is performed,
the minimum value after the quantization is represented by,
the maximum value after the quantization is represented,
representing the quantization factor at which the floating point value is to be accumulated in the graph volume.
Further, in step S4-4, the graph convolution operation is a matrix multiplication operation:
wherein, the first and the second end of the pipe are connected with each other,
representing the inverse quantized convolution output value of the graph,
representing the inverse-quantized activation value,
representing the inverse quantized graph convolution kernel,
a graph convolution operation is shown in a graph convolution operation,
representation diagramThe quantization factor of the convolved output values,
a quantized value representing the convolution output value of the graph,
is composed of
。
Quantization factor of floating point value of training set node characteristic based on characteristic aggregation layer
Quantization factor of the floating point value of the graph convolution kernel
Quantization factor of graph convolution output value
And/or the activation value after quantization
Quantized graph convolution kernel
Quantized value of the output value of the graph convolution
Obtaining the truncation bit width of the graph convolution output value
Obtaining the formula (8) through the formulas (2) - (7)
In the diagram convolution network quantification model based on the resistive random access memory, floating point values after the last diagram convolution inverse quantification are output through the softmax classifier.
Further, the loss function in step S4-5 is as shown in equations (9) - (12):
wherein
The node characteristic of the input training set passes through the output value of the ith neuron of the last layer of the graph convolution network model, C is the number of the output neurons, namely the category number of image classification,
representing an output value of the node characteristic of the input training set after passing through the softmax classifier;
wherein
A value of a tag representing the true value of the input image,
representing cross-entropy loss with the aim of reducing the error between the network output and the correct category of artificial labels;
wherein, the first and the second end of the pipe are connected with each other,
representing the number of current graph convolution layers in the graph convolution network,
representing the total number of layers of graph convolution in the graph convolution network,
represents the truncated bit width of the current graph convolution output value,
the target truncation bit width representing the convolution output value of the current graph is determined by the measuring range of the analog-to-digital converter;
optimizing the truncation bit width of the convolution output value of each layer of graph learned by the network to keep the truncation bit width consistent with the target truncation bit width;
wherein
Represent
The weight of (a) is calculated,
represent
The weight of (a) is calculated,
and
are all learnable parameters;
representing an overall loss function of the graph convolution network model; by optimising the loss functionAnd updating the weight parameters of the graph convolution network model, the quantization factors, the truncation bit width of the convolution output value of each layer of the graph, and the weight of the loss function until the network is converged.
Further, the step S6 includes the following steps:
step S6-1: firstly, for each document node of a test set, acquiring the characteristics of neighbor nodes of the node through a neighborhood characteristic extraction layer, and calculating the average value of the characteristics of the neighbor nodes and the characteristics of the neighbor nodes to be used as new characteristics of the node to obtain the new characteristics of all the nodes of the test set;
step S6-2: quantifying the floating point values of the new characteristics of all the test set nodes, then obtaining the quantified value of the activation quantification layer through activation operation, and mapping the quantified value into the voltage value of the resistive random access memory; quantizing the graph convolution kernel of the graph convolution quantization layer, and mapping the well-paid convolution kernel into a conductivity value of the resistive random access memory;
step 6-3: and performing graph convolution operation by adopting a resistive random access memory array, mapping the result of the graph convolution operation to a current value output by the resistive random access memory, converting the current value into a voltage value, sampling the voltage value through an analog-to-digital converter, taking the value obtained by sampling the quantized graph convolution as the input of the next layer of the network, and repeating the steps to obtain the prediction category of image classification according to the output value of the last layer of the graph convolution quantization layer.
Further, in the step S6-2, floating point values of all the node features of the test set are quantized according to the formula (13), and then the quantized values of the activated quantization layer are obtained through the ReLU activation operation
Will be
Mapping the voltage value V to a voltage value V, quantizing the graph convolution kernel of the graph convolution quantization layer learned by the network according to a formula (14) to obtain a quantized convolution kernel
Then will be
Mapping to a conductance value G based on the resistive random access memory;
wherein
A floating point value representing the characteristics of all test set nodes,
floating point values representing the graph convolution quantization layer graph convolution kernel,
a quantization factor representing the network learned active layer,
representing the quantization factors of the graph convolution layer of the network learned graph convolution kernel,
a quantized value representing an activated quantized layer,
representing the value of the graph convolution quantization layer after the convolution kernel quantization.
Graph convolution operation using resistive random access memory array
，
Recording the graph volume as a result of the graph convolution operationThe result of the product operation is mapped to a current value I output from the resistance change memory, the current value is converted into a voltage value,
sampling the voltage value through an analog-to-digital converter, and finally obtaining a value after convolution and quantization of the graph
Taking the maximum value index of the output value of the last graph convolution quantization layer as the type of network prediction;
wherein the content of the first and second substances,
representing the values of the graph after convolution quantization,
represents the truncated bit width of the graph convolution layer output value learned by the network,
a quantized value representing an activated quantized layer,
representing the value of the graph convolution quantization layer after the convolution kernel quantization.
Further, the floating point map convolution network model structure in step S2 is: input layer → neighborhood feature extraction layer → first convolution layer → first active layer → second convolution layer → second active layer → third convolution layer → softmax layer;
the structure of the graph convolution network quantization model in the step S3 is as follows: input layer → neighborhood feature extraction layer → first activation quantization layer → first activation inverse quantization layer → first graph volume inverse quantization layer → second activation quantization layer → second graph volume inverse quantization layer → second graph volume quantization layer → third activation quantization layer → third graph volume inverse quantization layer → softmax layer.
The structure of the graph convolution network quantization model in the step S5 is as follows: input layer → neighborhood feature extraction layer → first activation quantization layer → first map volume quantization layer → second activation quantization layer → second map volume quantization layer → third activation quantization layer → third map volume quantization layer → softmax layer.
The graph volume network document identification device based on the resistive random access memory comprises a memory and one or more processors, wherein executable codes are stored in the memory, and when the one or more processors execute the executable codes, the graph volume network document identification device based on the resistive random access memory is used for realizing the graph volume network document identification method based on the resistive random access memory.
The invention has the advantages and beneficial effects that:
aiming at specific hardware constraint, the invention learns the weight parameter, the quantization factor, the weight of the loss function and the truncation bit width of the convolution layer output value of each layer of the graph through a graph convolution network quantization model by combining a new loss function, the limited range of the conductance of the resistive random access memory and the fixed range of the analog-to-digital converter, so that the weight parameter, the quantization factor, the weight of the loss function and the truncation bit width of the convolution layer output value of each layer of the graph are kept consistent with the range of the analog-to-digital converter, and the precision loss caused by quantization is reduced. For the cora literature identification dataset, the quantization precision of 8 to 4 bits is almost lossless compared with the floating point precision.
Drawings
FIG. 1 is a flow chart of a method in an embodiment of the invention.
Fig. 2 is a partial network relationship diagram between documents in the embodiment of the present invention.
Fig. 3 is a diagram of a resistive random access memory crossbar array in an embodiment.
FIG. 4 is a schematic diagram of the apparatus of the present invention.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present invention, are given by way of illustration and explanation only, not limitation.
As shown in fig. 1, an example of an implementation of the present invention is the identification of a cora document identification data set. The documents in the data set have seven categories, and the categories of the documents are respectively based on cases, genetic algorithms, neural networks, probabilistic methods, reinforcement learning, rule learning and theories. Each document is cited or cited by at least one other document in the corpus. There are 2708 documents in the entire corpus, 5429 sides, and each document has 1433 unique words as features associated with the document. Fig. 2 shows a partial network relationship diagram between documents, and numbers in the diagram represent indexes of the documents.
The invention provides a literature identification device and method based on a resistive random access memory, which comprises the following steps:
step S1: and constructing a training set and a test set for the document identification data set, selecting 1208 documents as the training set, and randomly selecting 500 documents as the test set from the rest documents.
Step S2: constructing a floating point graph convolution network model based on a resistive random access memory, and performing model pre-training by using a training set to obtain pre-trained model parameters;
the specific floating point image convolution network model structure is as follows: input layer → neighborhood feature extraction layer → first convolution layer → first active layer → second convolution layer → second active layer → third convolution layer → softmax layer. And carrying out model pre-training by using the training set to obtain pre-trained model parameters. The size of the weight parameter for each layer is set as follows:
the size of the input layer is
；
A first graph convolution layer having a parameter of size
；
Second graph convolution layer, convolution kernel parameterThe size of the number is
；
A third graph convolution layer with convolution kernel parameters of size
And step S3: constructing a graph convolution network quantification model based on a training phase of a resistive random access memory according to the floating point graph convolution network model;
the specific graph convolution network quantization model structure is as follows: input layer → neighborhood feature extraction layer → first activation quantization layer → first activation inverse quantization layer → first graph volume inverse quantization layer → second activation quantization layer → second graph volume inverse quantization layer → second graph volume quantization layer → third activation quantization layer → third graph volume inverse quantization layer → softmax layer. The size of the weight parameter for each layer is set as follows:
the size of the input layer is
；
A first graph convolution quantization layer with a graph convolution kernel parameter of size
；
A second graph convolution quantization layer with convolution kernel parameters of size
；
A third graph convolution quantization layer with convolution kernel parameters of size
And step S4: inputting the training set into a graph convolution network quantization model in a training stage, and performing quantization perception training to obtain a truncation bit width of each layer of output values, a weight of a loss function and model parameters after quantization perception training;
the quantization bit width of the present embodiment is 8 bit,4 bit, 3 bit,2 bit. To [ -128,127] for 8-bit quantization, to [ -8,7] for 4-bit quantization, to [ -4,3] for 3-bit quantization, and to [ -2,1] for 2-bit quantization. The method comprises the following specific steps:
step S4-1: aggregating information of neighbor nodes through a feature aggregation layer for each document node of the training set;
the specific polymerization mode is as follows: acquiring the characteristics of a fixed number of first-order neighbor nodes of the node in a random sampling mode, cascading the characteristics with the characteristics of the node, then calculating the mean value of the cascaded characteristics in each characteristic dimension to be used as a new characteristic of the node, and obtaining the characteristic values of all training set nodes in a characteristic aggregation layer; the aggregation function is shown in equation (1):
wherein, the first and the second end of the pipe are connected with each other,
representing a target node
The input characteristic value of (2) is,
first order neighbor node representing target node
The input characteristic value of (2) is,
meaning that the target node is concatenated with the input eigenvalues of all first-order neighbor nodes,
representing a target node
The characteristic value of the characteristic aggregation layer.
Step S4-2: the characteristic values of all the training set nodes obtained in the step S4-1 in the characteristic aggregation layer are quantized through the activation quantization layer to obtain quantized activation values
；
As shown in formula (2), performing inverse quantization operation by the first active inverse quantization layer, and obtaining the active value after inverse quantization by the ReLU activation function
As shown in equation (3);
wherein
Floating-point eigenvalues representing all training set nodes at the feature aggregation level,
it is meant to round-off the process,
it is indicated that the operation of truncation is performed,
which represents the minimum value after the quantization and,
representing the maximum value after quantization.
Expressing trainingAnd refining the quantization factor of the floating point value of the node characteristic.
Step S4-3: performing quantization operation on the graph convolution kernel through a graph convolution quantization layer to obtain a quantized graph convolution kernel, and performing inverse quantization operation on the graph convolution kernel through a graph convolution inverse quantization layer to obtain an inverse quantized graph convolution kernel;
obtaining the graph convolution kernel after quantization as shown in formula (4)
Performing inverse quantization operation by the first graph convolution inverse quantization layer, as shown in formula (5), to obtain graph convolution kernel after inverse quantization
；
Step S4-4: inverse quantizing the activation value
And inverse quantized graph convolution kernel
Performing graph convolution operation to obtain inverse quantized graph convolution output value
，
Representing graph convolution kernelsThe quantization factor of (a);
as shown in formula (6), wherein
Representing graph convolution operations, i.e., matrix multiplication operations;
equation (8) can be derived from equations (2) - (7), wherein,
a quantization factor representing the output value of the graph convolution,
a quantized value representing the output value of the graph convolution,
is composed of
，
Representing the truncated bit width of the first graph convolution output value.
And obtaining the truncation bit width of the graph convolution output value based on the quantization factor and the quantization value, and in the same way, inversely quantizing the last graph convolution by using the floating point value, and obtaining the output of the graph convolution network quantization model in the training stage through the classifier.
Step S4-5: and updating network parameters including quantization factors of each layer, truncation bit width of convolution output values of each layer of the graph and weight of the loss function by optimizing the loss function until the network converges to obtain a graph convolution network quantization model after quantization perception training.
The specific loss functions are shown in equations (9) - (12):
wherein
The node characteristic representing the input training set passes through the output value of the ith neuron of the last layer of the graph convolution network model, C is the number of the output neurons, namely the classified category number,
and (4) representing the output value of the node feature of the input training set after passing through the softmax classifier.
Wherein
A value of a tag representing the true value of the input image,
represents cross-entropy loss with the goal of reducing the error between the network output and the correct category of artificial labeling.
Wherein, the first and the second end of the pipe are connected with each other,
representing the number of layers of the current graph convolution in the graph convolution network,
representing the total number of layers of graph convolution in the graph convolution network,
represents the truncated bit width of the current graph convolution output value,
the target truncation bit width representing the convolution output value of the current diagram is determined by the range of the analog-to-digital converter.
The truncation bit width of the convolution output value of each layer of graph learned by the network is optimized, so that the truncation bit width is consistent with the target truncation bit width.
Wherein
To represent
The weight of (a) is calculated,
to represent
The weight of (a) is calculated,
and
are all learnable parameters.
The overall loss function of the graph convolution network model is represented. By optimizing the lossAnd (4) losing functions, updating weight parameters and quantization factors of the graph convolution network model, and truncating bit width of each layer of graph convolution output values until the network is converged.
Step S5: constructing a graph convolution network quantization model of an inference stage based on the resistive random access memory according to the graph convolution network quantization model of the training stage;
the specific graph convolution network quantization model structure is as follows: the input layer → the neighborhood feature extraction layer → the first activation quantization layer → the first volume quantization layer → the second activation quantization layer → the second volume quantization layer → the third activation quantization layer → the third volume quantization layer → the softmax layer.
Step S6: and mapping the model parameters after the quantitative perception training to the resistive random access memory, inputting the test set to a graph convolution network quantitative model based on an inference stage of the resistive random access memory, and performing forward inference test.
As shown in fig. 3, V represents a voltage value, G represents a conductance value, and I represents a current value, and the specific steps are as follows:
step S6-1: firstly, for each document node of a test set, acquiring the characteristics of neighbor nodes of the node through a neighborhood characteristic extraction layer, and calculating the average value of the characteristics of the neighbor nodes and the characteristics of the neighbor nodes to be used as new characteristics of the node to obtain the new characteristics of all the nodes of the test set;
step S6-2: quantizing the floating point values of the new characteristics of all the nodes of the test set, then obtaining a quantized value of an activation quantization layer through activation operation, and mapping the quantized value into a voltage value of the resistive random access memory; quantizing the graph convolution kernel of the graph convolution quantization layer, and mapping the well-paid convolution kernel into a conductivity value of the resistive random access memory;
specifically, the new features of all the nodes in the test set pass through a first activation quantization layer and a first graph convolution quantization layer to obtain a value after convolution quantization of a first graph
. The quantization method is shown in equations (13) to (15):
wherein
A floating point value representing the characteristics of all test set nodes,
a floating point value representing a convolution kernel of the first graph volume quantization layer graph,
a quantization factor representing the first active layer learned by the network,
a quantization factor representing a first graph convolution layer learned by the network,
representing the quantized values of the first activated quantization layer.
Representing the quantized values of the first map convolution quantization layer map convolution kernel.
Quantizing the floating point values of all the node characteristics of the test set according to a formula (13), and then obtaining the quantized value of the first activation quantization layer through the ReLU activation operation
Will be
Mapped to a voltage value. Quantizing the graph convolution kernel of the first graph convolution quantization layer learned by the network according to the formula (14) to obtain a quantized convolution kernel
. Then will be
The mapping is based on the conductance value of the resistive random access memory.
Step 6-3: and performing graph convolution operation by adopting a resistive random access memory array, mapping the result of the graph convolution operation to a current value output by the resistive random access memory, converting the current value into a voltage value, sampling the voltage value through an analog-to-digital converter, taking the value obtained by sampling the quantized graph convolution as the input of the next layer of the network, and repeating the steps to obtain the prediction category of image classification according to the output value of the last layer of the graph convolution quantization layer.
Specifically, a resistive random access memory array is used for graph convolution operation
，
Is recorded as the result of the graph convolution operation. And mapping the result of the graph convolution operation to a current value output by the resistive random access memory, and converting the current value into a voltage value.
Sampling the voltage value through an analog-to-digital converter to obtain a value after convolution and quantization of a first graph
As input to the next layer of the network. By parity of reasoning, obtainAnd taking the index of the maximum value of the output value of the last layer of graph convolution quantization layer as the category of the network prediction.
The effect of the present invention is further explained by combining the simulation experiment as follows:
1. simulation conditions are as follows:
the simulation experiment of the invention is carried out under the hardware environment of NVIDIA GV100 and the software environment of Pytrch 1.5.
2. Simulation content and result analysis:
the invention carries out literature identification on the cora literature identification data set through the graph convolution network. The invention respectively uses a graph convolution floating point model, an 8 bit quantization model, a 4 bit quantization model, a 3 bit quantization model and a 2 bit quantization model to identify the cora test set. Table 1 shows the average recognition accuracy of the test set and the truncation bit width of the output value of the first layer graph convolution layer after the convergence of the network model
Truncation bit width of the second layer map convolution layer output value
Loss function
Weight of (2)
Loss function
Weight of (2)
. The range of the analog-to-digital converter of the present embodiment is
I.e., 0.03125, it can be seen that after the above network model converges,
and
very close to the range of the analog-to-digital converter. Meanwhile, the average test precision of the 8-bit quantization model and the average test precision of the 4-bit quantization model are basically consistent with the average test precision of the floating point model, the average test precision of the 3-bit quantization model is 3% lower than the average test precision of the floating point model, and the average test precision of the 2-bit quantization model is 8.9% lower than the average test precision of the floating point model, so that the loss of the quantization precision is low when the quantization bit width is 3-8 bits, and the method can achieve good effect on document identification application of the core data set.
Table 1 comparison table of average identification precision of floating point model and low bit quantization model to test set
In summary, the invention provides a graph convolution network document identification device and method based on a resistive random access memory, which are applied to various fields such as text classification, recommendation systems, knowledge graph spectrum compensation and the like, aiming at specific hardware constraints. The method combines the characteristics of limited conductance range of the resistive random access memory and fixed range of the analog-to-digital converter, and learns the weight parameters, the quantization factors, the weight of the loss function and the truncation bit width of the output value of each layer of graph convolution layer through the graph convolution network model, so that the weight parameters, the quantization factors, the weight of the loss function and the truncation bit width of each layer of graph convolution layer are consistent with the range of the analog-to-digital converter, and the precision loss caused by quantization is reduced. For the cora literature identification data set, 3 bit to 8 bit quantization models can achieve good effects.
Corresponding to the embodiment of the graph convolution network document identification method based on the resistive random access memory, the invention also provides an embodiment of a graph convolution network document identification device based on the resistive random access memory.
Referring to fig. 4, the apparatus for identifying a literature of a resistance change memory-based graph volume network provided by an embodiment of the present invention includes a memory and one or more processors, where the memory stores executable codes, and when the one or more processors execute the executable codes, the apparatus is configured to implement the method for identifying a literature of a resistance change memory-based graph volume network in the above embodiment.
The embodiment of the graph convolution network document identification device based on the resistive random access memory can be applied to any equipment with data processing capability, such as computers and other equipment or devices. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. The software implementation is taken as an example, and as a logical device, the device is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory for running through the processor of any device with data processing capability. In terms of hardware, as shown in fig. 4, the diagram is a hardware structure diagram of an arbitrary device with data processing capability where the apparatus is located based on the graph convolution network document identification of the resistive random access memory according to the present invention, and besides the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 4, the arbitrary device with data processing capability where the apparatus is located in the embodiment may also include other hardware according to the actual function of the arbitrary device with data processing capability, which is not described again.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the invention. One of ordinary skill in the art can understand and implement without inventive effort.
The embodiment of the invention also provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the method for identifying the literature of the graph volume network based on the resistive random access memory in the above embodiment is implemented.
The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any data processing capability device described in any of the foregoing embodiments. The computer readable storage medium may also be any external storage device of a device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD Card, a Flash memory Card (Flash Card), etc. provided on the device. Further, the computer readable storage medium may include both an internal storage unit and an external storage device of any data processing capable device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing capable device, and may also be used for temporarily storing data that has been output or is to be output.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A graph convolution network document identification method based on a resistive random access memory is characterized by comprising the following steps:
step S1: constructing a training set and a test set for a document identification data set;
step S2: constructing a floating point graph convolution network model based on a resistive random access memory, and performing model pre-training by using a training set to obtain pre-trained model parameters;
and step S3: constructing a graph convolution network quantification model based on a training phase of the resistive random access memory according to the floating point graph convolution network model;
and step S4: inputting the training set into a graph convolution network quantization model in a training stage, and performing quantization perception training to obtain a truncation bit width of each layer of output values, a weight of a loss function and model parameters after quantization perception training;
step S5: constructing a graph convolution network quantization model of an inference stage based on the resistive random access memory according to the graph convolution network quantization model of the training stage;
step S6: and mapping the model parameters after the quantitative perception training to the resistive random access memory, inputting the test set to a graph convolution network quantitative model based on an inference stage of the resistive random access memory, and performing forward inference test.
2. The graph convolution network document identification method based on the resistive random access memory according to claim 1, characterized in that: the step S4 includes the steps of:
step S4-1: for each document node of the training set, aggregating the information of neighbor nodes through a feature aggregation layer;
step S4-2: performing quantization operation on the characteristic values of all the training set nodes in the characteristic aggregation layer obtained in the step S4-1 through an activation quantization layer to obtain activation values after quantization;
step S4-3: carrying out quantization operation on the graph convolution kernel through a graph convolution quantization layer to obtain a quantized graph convolution kernel, and carrying out inverse quantization operation on the graph convolution kernel through a graph convolution inverse quantization layer to obtain an inverse quantized graph convolution kernel;
step S4-4: carrying out the graph convolution operation on the activation value after the inverse quantization and the graph convolution kernel after the inverse quantization to obtain a graph convolution output value after the inverse quantization; obtaining the truncation bit width of the graph convolution output value based on the quantization factor and the quantization value, and in the same way, inversely quantizing the last graph convolution and then obtaining the output of the graph convolution network quantization model in the training stage through a classifier;
step S4-5: and updating network parameters including quantization factors of each layer, truncation bit width of a convolution output value of each layer and weight of the loss function by optimizing the loss function until the network converges to obtain a convolution network quantization model after quantization perception training.
3. The method for identifying the graph convolution network literature based on the resistive random access memory according to claim 2, wherein the method comprises the following steps: the polymerization mode in the step S4-1 is as follows: acquiring the characteristics of a fixed number of first-order neighbor nodes of the node in a random sampling mode, cascading the characteristics with the characteristics of the node, then calculating the mean value of the cascaded characteristics in each characteristic dimension to be used as a new characteristic of the node, and obtaining the characteristic values of all training set nodes in a characteristic aggregation layer; the aggregation function is as follows:
wherein, the first and the second end of the pipe are connected with each other,
representing a target node
The input characteristic value of (2) is,
first order neighbor node representing target node
The input characteristic value of (2) is,
meaning that the target node is concatenated with the input eigenvalues of all first order neighbor nodes,
representing a target node
The characteristic value of the characteristic aggregation layer.
4. The graph convolution network document identification method based on the resistive random access memory according to claim 2, characterized in that: the activation value after quantization in step S4-2:
then, carrying out inverse quantization operation by activating an inverse quantization layer, and obtaining an activation value after inverse quantization by a ReLU activation function:
wherein
Representing the floating-point eigenvalues of all training set nodes at the feature aggregation level,
it is meant to round-off the process,
it is indicated that the operation of truncation is performed,
which represents the minimum value after the quantization and,
the maximum value after the quantization is represented,
a quantization factor representing a floating point value of a training set node feature at a feature aggregation layer;
the graph convolution kernel after quantization in the step S4-3:
the graph convolution kernel after the dequantization:
wherein
Representing the graph convolution kernel floating point values,
it is meant to round-off the process,
it is indicated that the operation of truncation is performed,
the minimum value after the quantization is represented by,
the maximum value after the quantization is represented,
representing the quantization factor at which the floating point value is rolled up in the graph.
5. The graph convolution network document identification method based on the resistive random access memory according to claim 2, characterized in that: in step S4-4, the graph convolution operation is a matrix multiplication operation:
wherein the content of the first and second substances,
representing the inverse quantized convolution output value of the graph,
representing the inverse-quantized activation value,
representing the inverse quantized graph convolution kernel,
a graph convolution operation is shown in a graph convolution operation,
a quantization factor representing the output value of the graph convolution,
a quantized value representing the output value of the graph convolution,
is composed of
；
Quantization factor of floating point value of training set node characteristic based on characteristic aggregation layer
Quantization factor of the floating point value of the graph convolution kernel
Quantization factor of graph convolution output value
And/or the activation value after quantization
Quantized graph convolution kernel
Quantized value of the convolution output value
Obtaining the truncation bit width of the output value of the graph convolution
Obtaining the formula (8) through the formulas (2) - (7)
In the diagram convolution network quantification model based on the resistive random access memory, floating point values after the last diagram convolution inverse quantification are output through the softmax classifier.
6. The graph convolution network document identification method based on the resistive random access memory according to claim 2, characterized in that: the loss function in step S4-5 is shown in equations (9) - (12):
wherein, the node characteristic of the input training set passes through the output value of the ith neuron of the last layer of the graph convolution network model, C is the number of the output neurons, namely the class number of the image classification,
representing an output value of the node characteristic of the input training set after passing through the softmax classifier;
wherein
A value of a tag representing the true value of the input image,
represents the cross entropy loss;
wherein, the first and the second end of the pipe are connected with each other,
representing the number of layers of the current graph convolution in the graph convolution network,
representing the total number of layers of graph convolution in the graph convolution network,
represents the truncated bit width of the current graph convolution output value,
a target truncation bit width representing a current graph convolution output value;
optimizing the truncation bit width of the convolution output value of each layer of graph learned by the network to keep the truncation bit width consistent with the target truncation bit width;
wherein
To represent
The weight of (a) is calculated,
to represent
The weight of (a) is calculated,
and
are all learnable parameters;
representing an overall loss function of the graph convolution network model; by optimizing the loss function, the weight parameter, the quantization factor, the truncation bit width of the convolution output value of each layer of the graph and the weight of the loss function are updated until the network is converged.
7. The graph convolution network document identification method based on the resistive random access memory according to claim 1, characterized in that: in the step S6, the method includes the following steps:
step S6-1: firstly, for each document node of a test set, acquiring the characteristics of neighbor nodes of the node through a neighborhood characteristic extraction layer, and calculating the average value of the characteristics of the neighbor nodes and the characteristics of the neighbor nodes to be used as new characteristics of the node to obtain the new characteristics of all the nodes of the test set;
step S6-2: quantizing the floating point values of the new characteristics of all the nodes of the test set, then obtaining a quantized value of an activation quantization layer through activation operation, and mapping the quantized value into a voltage value of the resistive random access memory; quantizing a graph convolution kernel of a graph convolution quantization layer, and mapping the quantized convolution kernel to a conductance value of the resistive random access memory;
step 6-3: the method comprises the steps of performing graph convolution operation by adopting a resistive random access memory array, mapping a result of the graph convolution operation to a current value output by the resistive random access memory, converting the current value into a voltage value, sampling the voltage value through an analog-to-digital converter, taking a value obtained by sampling after graph convolution quantization as input of a next layer of a network, and repeating the steps in sequence to obtain a prediction type of image classification according to an output value of a last layer of graph convolution quantization layer.
8. The graph convolution network document identification method based on the resistive random access memory is characterized in that: in the step S6-2, floating point values of all the node characteristics of the test set are quantized according to a formula (13), and then the quantized values of the activated quantization layer are obtained through ReLU activation operation
Will be
Mapping the voltage value V to a voltage value V, quantizing the graph convolution kernel of the graph convolution quantization layer learned by the network according to a formula (14) to obtain a quantized convolution kernel
Then will be
Mapping to a conductance value G based on the resistive random access memory;
wherein
A floating point value representing the characteristics of all test set nodes,
floating point values representing the graph convolution quantization layer graph convolution kernel,
a quantization factor representing the network learned active layer,
representing the quantization factors of the network learned graph convolution kernel,
representing the quantized values of the active quantization layer,
representing a value quantized by a graph convolution quantization layer graph convolution kernel;
graph convolution operation using resistive random access memory array
，
Mapping the result of the graph convolution operation to a current value I output from the resistance change memory, converting the current value into a voltage value,
sampling the voltage value through an analog-to-digital converter, and finally obtaining a value after convolution and quantization of the graph
Obtaining the quantization value of the convolution of the last layer of the graph by analogy as the input of the next layer of the network, and taking the index of the maximum value of the output value of the convolution quantization layer of the last layer of the graph as the category of the network prediction;
wherein, the first and the second end of the pipe are connected with each other,
representing the values of the graph after convolution quantization,
indicating the truncation bit width of the graph convolution layer output value learned by the network,
a quantized value representing an activated quantized layer,
representing the value of the graph after the graph convolution quantization layer graph convolution kernel quantization.
9. The graph convolution network document identification method based on the resistive random access memory according to claim 1, characterized in that:
the floating point image convolution network model structure in the step S2 is connected in sequence: the system comprises an input layer, a neighborhood feature extraction layer, a first graph volume layer, a first activation layer, a second graph volume layer, a second activation layer, a third graph volume layer and a softmax layer;
the graph convolution network quantization model structure in the step S3 is connected in sequence: the system comprises an input layer, a neighborhood feature extraction layer, a first activation quantification layer, a first activation inverse quantification layer, a first graph volume inverse quantification layer, a second activation inverse quantification layer, a second graph volume inverse quantification layer, a third activation inverse quantification layer, a third graph volume inverse quantification layer and a softmax layer;
the graph convolution network quantization model structure in the step S5 is sequentially connected: the system comprises an input layer, a neighborhood feature extraction layer, a first activation quantification layer, a first graph volume quantification layer, a second activation quantification layer, a second graph volume quantification layer, a third activation quantification layer, a third graph volume quantification layer and a softmax layer.
10. A resistance change memory-based graph volume network document identification device, comprising a memory and one or more processors, wherein the memory stores executable codes, and the one or more processors execute the executable codes to implement the resistance change memory-based graph volume network document identification method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310017218.7A CN115905546B (en) | 2023-01-06 | 2023-01-06 | Graph convolution network literature identification device and method based on resistive random access memory |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310017218.7A CN115905546B (en) | 2023-01-06 | 2023-01-06 | Graph convolution network literature identification device and method based on resistive random access memory |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115905546A true CN115905546A (en) | 2023-04-04 |
| CN115905546B CN115905546B (en) | 2023-07-14 |
Family
ID=86495677
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310017218.7A Active CN115905546B (en) | 2023-01-06 | 2023-01-06 | Graph convolution network literature identification device and method based on resistive random access memory |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115905546B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190340492A1 (en) * | 2018-05-04 | 2019-11-07 | Microsoft Technology Licensing, Llc | Design flow for quantized neural networks |
| CN110852439A (en) * | 2019-11-20 | 2020-02-28 | 字节跳动有限公司 | Neural network model compression and acceleration method, data processing method and device |
| US20200293889A1 (en) * | 2018-03-06 | 2020-09-17 | Tdk Corporation | Neural network device, signal generation method, and program |
| CN112215331A (en) * | 2019-07-10 | 2021-01-12 | 华为技术有限公司 | Data processing method for neural network system and neural network system |
| CN112598123A (en) * | 2020-12-25 | 2021-04-02 | 清华大学 | Weight quantization method and device of neural network and storage medium |
| CN113516234A (en) * | 2021-05-10 | 2021-10-19 | 西安交通大学 | Method and device for relieving nonideal factors of memristor accelerator |
| CN114646839A (en) * | 2022-01-28 | 2022-06-21 | 国网河北省电力有限公司保定供电分公司 | Power distribution network fault section positioning method and device based on graph convolution neural network |
| CN114677548A (en) * | 2022-05-26 | 2022-06-28 | 之江实验室 | Neural network image classification system and method based on resistive random access memory |
| CN115310595A (en) * | 2022-08-11 | 2022-11-08 | 中国人民解放军国防科技大学 | Neural network mixing precision quantification method and system of memristor |
| CN115311506A (en) * | 2022-10-11 | 2022-11-08 | 之江实验室 | Image classification method and device based on quantization factor optimization of resistive random access memory |
-
2023
- 2023-01-06 CN CN202310017218.7A patent/CN115905546B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200293889A1 (en) * | 2018-03-06 | 2020-09-17 | Tdk Corporation | Neural network device, signal generation method, and program |
| US20190340492A1 (en) * | 2018-05-04 | 2019-11-07 | Microsoft Technology Licensing, Llc | Design flow for quantized neural networks |
| CN112215331A (en) * | 2019-07-10 | 2021-01-12 | 华为技术有限公司 | Data processing method for neural network system and neural network system |
| CN110852439A (en) * | 2019-11-20 | 2020-02-28 | 字节跳动有限公司 | Neural network model compression and acceleration method, data processing method and device |
| CN112598123A (en) * | 2020-12-25 | 2021-04-02 | 清华大学 | Weight quantization method and device of neural network and storage medium |
| WO2022135209A1 (en) * | 2020-12-25 | 2022-06-30 | 清华大学 | Quantization method and quantization apparatus for weight of neural network, and storage medium |
| CN113516234A (en) * | 2021-05-10 | 2021-10-19 | 西安交通大学 | Method and device for relieving nonideal factors of memristor accelerator |
| CN114646839A (en) * | 2022-01-28 | 2022-06-21 | 国网河北省电力有限公司保定供电分公司 | Power distribution network fault section positioning method and device based on graph convolution neural network |
| CN114677548A (en) * | 2022-05-26 | 2022-06-28 | 之江实验室 | Neural network image classification system and method based on resistive random access memory |
| CN115310595A (en) * | 2022-08-11 | 2022-11-08 | 中国人民解放军国防科技大学 | Neural network mixing precision quantification method and system of memristor |
| CN115311506A (en) * | 2022-10-11 | 2022-11-08 | 之江实验室 | Image classification method and device based on quantization factor optimization of resistive random access memory |
Non-Patent Citations (2)
| Title |
|---|
| WANG MENG: "Study on methods of suppressing signal truncation error in digital signal processing", 《2019 14TH IEEE INTERNATIONAL CONFERENCE ON ELECTRONIC MEASUREMENT & INSTRUMENTS(ICEMI)》 * |
| 孙磊;肖金球;夏禹;顾敏明;: "改进的基于嵌入式SoC卷积神经网络识别模型", 计算机应用与软件, no. 03 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115905546B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Carreira-Perpinán | Model compression as constrained optimization, with application to neural nets. Part I: General framework | |
| US11544542B2 (en) | Computing device and method | |
| CN113837370B (en) | Method and apparatus for training a model based on contrast learning | |
| US11537879B2 (en) | Neural network weight discretizing method, system, device, and readable storage medium | |
| CN114677548B (en) | Neural network image classification system and method based on resistive random access memory | |
| CN115311506B (en) | Image classification method and device based on quantization factor optimization of resistive random access memory | |
| Moya Rueda et al. | Neuron pruning for compressing deep networks using maxout architectures | |
| WO2023124342A1 (en) | Low-cost automatic neural architecture search method for image classification | |
| Rani et al. | DeepCompNet: A novel neural net model compression architecture | |
| EP3882823A1 (en) | Method and apparatus with softmax approximation | |
| Bao et al. | Cross-entropy pruning for compressing convolutional neural networks | |
| CN112949610A (en) | Improved Elman neural network prediction method based on noise reduction algorithm | |
| CN113378866B (en) | Image classification method, system, storage medium and electronic device | |
| CN115905546A (en) | Graph convolution network document identification device and method based on resistive random access memory | |
| US20220253709A1 (en) | Compressing a Set of Coefficients for Subsequent Use in a Neural Network | |
| CN115599918A (en) | Mutual learning text classification method and system based on graph enhancement | |
| Goel et al. | CompactNet: High accuracy deep neural network optimized for on-chip implementation | |
| CN114595641A (en) | Method and system for solving combined optimization problem | |
| CN114819140A (en) | Model pruning method and device and computer equipment | |
| CN112988548A (en) | Improved Elman neural network prediction method based on noise reduction algorithm | |
| Hossain et al. | Computational Complexity Reduction Techniques for Deep Neural Networks: A Survey | |
| CN113361707A (en) | Model compression method, system and computer readable medium | |
| CN112651492A (en) | Self-connection width graph convolution neural network model and training method thereof | |
| CN116737607B (en) | Sample data caching method, system, computer device and storage medium | |
| Dong et al. | An optimization method for pruning rates of each layer in CNN based on the GA-SMSM |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |