CN110555340A - neural network computing method and system and corresponding dual neural network implementation - Google Patents
neural network computing method and system and corresponding dual neural network implementation Download PDFInfo
- Publication number
- CN110555340A CN110555340A CN201810550516.1A CN201810550516A CN110555340A CN 110555340 A CN110555340 A CN 110555340A CN 201810550516 A CN201810550516 A CN 201810550516A CN 110555340 A CN110555340 A CN 110555340A
- Authority
- CN
- China
- Prior art keywords
- neural network
- feature
- class
- features
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N99/00—Subject matter not provided for in other groups of this subclass
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
A neural network computing method, system and corresponding dual network implementation are provided. The calculation method comprises the following steps: obtaining input for neural network computations, the input comprising features of two or more categories; extracting a first feature of the input for the first class based on a first class neural network; extracting a second reference feature of the input for a second class based on a second class reference neural network; introducing the second reference feature into the first feature to remove the effect of a second one of the first features; and completing neural network classification calculation for the first feature based on the first feature superimposed with the second reference feature. According to the invention, before the feature vectors obtained through the neural network are sent to the classifier, the feature vectors are superposed with the relevant reference feature vectors to remove the distribution influence of the relevant features on the target features, so that the classification accuracy is improved.
Description
Technical Field
The invention relates to the field of deep learning, in particular to a neural network computing method and system and a corresponding dual neural network implementation.
background
Methods based on Artificial Neural Networks (ANN), in particular Convolutional Neural Networks (CNN), have been very successful in many applications. In the field of computer vision, particularly aiming at the problem of image classification, the introduction of the CNN greatly improves the precision of image classification.
For example, the application of deep learning using CNN to face recognition is very successful, and the history of the deep learning is new and high in many data sets, but the problem of interpretability still exists. Because the improvement on the objective function is less and less obvious for improving the accuracy of face recognition, how to further reduce the class inner distance becomes a problem to be solved urgently.
Disclosure of Invention
in order to solve at least one of the above problems, the present invention proposes a neural network computing scheme that removes the distribution influence of relevant features on target features by superimposing a feature vector obtained via a neural network on a relevant reference feature vector before feeding the feature vector into a classifier, thereby improving classification accuracy.
According to an aspect of the present invention, a neural network computing method is provided, including: obtaining input for neural network computations, the input comprising features of two or more categories; extracting a first feature of the input for the first class based on a first class neural network; extracting a second reference feature of the input for a second class based on a second class reference neural network; introducing the second reference feature into the first feature to remove the effect of a second one of the first features; and completing neural network classification calculation for the first feature based on the first feature superimposed with the second reference feature.
Therefore, the influence of other features on the target feature can be removed through the introduction of the reference feature, and the classification accuracy of the target feature is improved.
the method may further comprise: extracting a second feature of the input for a second category based on a second category neural network; extracting a first reference feature of the input for the first class based on a first class reference neural network; introducing the first reference feature into the second feature to remove the effect of the first one of the second features; and completing neural network classification calculation for the second feature based on the second feature superimposed with the first reference feature. Thereby providing a more accurate joint target dual task classification calculation.
The second reference feature may be a second feature not containing the first category of feature information and/or the first reference feature may be a first feature not containing the second category of feature information. Thereby further enhancing the purity of the reference feature.
The second class of reference neural network may be a side branch neural network of the second class of neural network, and/or the first class of reference neural network may be a side branch neural network of the first class of neural network. Therefore, by introducing the side branch network, the network efficiency is improved and the network arrangement cost is reduced. The weights and biases of the side branch neural network are trained based on an objective function that competes against the first and second features. Preferably, the objective function of the countermeasure is achieved by inserting a gradient inversion layer. Thereby removing the influence of other features on the reference feature.
in one implementation, the input may be an image, the first class of neural network is a face recognition neural network, the first feature is a face feature, the second class of neural network is a face attribute recognition neural network, and the second feature is a face attribute feature.
The second category may include a plurality of other categories of the two or more categories of features excluding the first category to accommodate situations in which three or more related features are included to influence the intra-category distribution.
Preferably, extracting a second reference feature of the input for a second class based on a second class reference neural network may include: extracting a plurality of second reference features based on the plurality of second class reference neural networks; and introducing the second reference feature into the first feature to remove the effect of the second one of the first features may comprise: a plurality of second reference features are respectively introduced into the first features to remove the effect of each of the second features in the first features.
Extracting a second reference feature of the input for a second class based on a second class reference neural network may also include: extracting a second reference feature fusing the other multiple class features based on a single second class reference neural network fusing the other multiple classes.
Preferably, completing the neural network computation for the first feature based on the first feature on which the second reference feature is superimposed comprises: and sending the first feature superposed with the second reference feature into a classifier of the first-class neural network to obtain a classification result.
According to another aspect of the present invention, there is provided a neural network computing system, comprising: a first class neural network for extracting a first feature for a first class from an input; a second class of reference neural network for extracting a second reference feature for a second class from the input, wherein the first class of neural network further comprises an overlay module for introducing the second reference feature into the first feature to remove an influence of a second feature in the first feature, and the first class of neural network completes a neural network classification calculation for the first feature based on the first feature on which the second reference feature is overlaid.
The system may further comprise: a second class neural network for extracting second features for a second class from the input; the second class neural network further comprises an overlapping module, the overlapping module introduces the first reference feature into the second feature to remove the influence of the first feature in the second feature, and the second class neural network completes the neural network classification calculation for the second feature based on the second feature overlapped with the first reference feature.
The first class of reference neural network may be a side branch neural network of the first class of neural network, and the second class of reference neural network may be a side branch neural network of the second class of neural network. The weights and biases of the side branch neural network may be trained based on an objective function that competes against the first and second features, and the obtained reference features are features that do not contain other class feature information.
Preferably, the input may be an image, the first class of neural network is a face recognition neural network, the first features are face features, and the second class of neural network is a face attribute recognition neural network, the second features are face attribute features.
the second category may include a plurality of other categories of the features of the two or more categories excluding the first category, and wherein: the second class of reference neural networks is a plurality of second class of reference neural networks each based on one of the other plurality of classes, and/or a single second class of reference neural network based on the other plurality of classes.
According to yet another aspect of the present invention, there is provided a dual neural network for joint calculation, including: a face recognition neural network as a backbone neural network for extracting face features from the input image; a face attribute recognition neural network as another backbone neural network for extracting a face attribute feature from the input image; a face recognition reference neural network serving as a side branch network of the face recognition neural network extracts face reference features which do not contain face attribute information from the input image; the face attribute recognition neural network is used as a side branch network of the face attribute recognition neural network, and face attribute reference features which do not contain face information are extracted from the input image, wherein the face attribute recognition neural network comprises an overlapping module, the overlapping module introduces the face attribute reference features into the face features to remove the influence of the face attribute features in the face features, and the face attribute recognition neural network completes neural network calculation aiming at the face features based on the face features overlapped with the face attribute reference features.
The side branch neural network and the main neural network share most of the preceding network structure, and the reference features are solved based on weights and bias parameters obtained by training an objective function for confrontation of the face and the face attribute features.
According to an aspect of the present invention, there is provided a computing device comprising: a processor; and a memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the neural network computing method of any one of the above.
According to yet another aspect of the invention, there is provided a non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform a neural network computing method as claimed in any one of the above.
According to the neural network computing scheme and the double networks thereof, before the feature vectors obtained through the neural network are sent to the classifier, the distribution influence of the relevant features on the target features is removed through superposition with the relevant reference feature vectors, and therefore the classification accuracy is improved. Furthermore, through cross countermeasure learning of the two networks, the class internal distance of the extracted features of the two networks is reduced, so that information irrelevant to the main task is less, and the accuracy of relevant tasks such as face recognition and face attribute recognition tasks is improved.
Drawings
the above and other objects, features and advantages of the present disclosure will become more apparent by describing in greater detail exemplary embodiments thereof with reference to the attached drawings, in which like reference numerals generally represent like parts throughout.
Fig. 1 shows a typical CNN layer composition example.
FIG. 2 shows a flow diagram of a neural network computational method, according to one embodiment of the invention.
Fig. 3 shows the relevant steps of a preferred sub-scheme of the neural network computing method according to the present invention.
FIG. 4 shows a schematic diagram of a neural network computing system, in accordance with one embodiment of the present invention.
Fig. 5 shows an example of feature superposition in the feature space.
FIG. 6 illustrates a schematic diagram of a neural network computing system, in accordance with another embodiment of the present invention.
FIG. 7 shows a schematic diagram of a dual neural network implementation, according to one embodiment of the present invention.
Fig. 8 is a schematic structural diagram of a data processing computing device that can be used to implement the neural network computing method according to an embodiment of the present invention.
Detailed Description
Preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The traditional face recognition adopts a characteristic engineering method, image characteristics are designed manually to classify face pictures, and the method has large engineering quantity and poor adaptability. The deep learning method gradually becomes a mainstream method in the field of computer vision from 2012, and a deep artificial neural network represented by a convolutional neural network has the effect far beyond that of a traditional method in image classification, detection, tracking, segmentation and other tasks. The neural network can automatically extract the effective characteristics of the task, so that the deep learning method avoids complex characteristic engineering. Deep learning, an important component of differentiable programming, has become the most popular approach throughout the field of artificial intelligence. The human face recognition is the field that computer vision is firstly applied to the industry, human face features are extracted through a deep neural network, similarity comparison is carried out on the extracted human face features and human faces in a database, and the human faces are judged to be in the database or not in the database, so that the task of human face recognition is completed.
Although the application of deep learning to face recognition is very successful, and the history is new and high on many data sets, the problems of interpretability and the like still exist. The improvement on the objective function is less and less obvious for improving the face recognition accuracy, and how to further reduce the class inner distance becomes a problem to be solved urgently. The inventor of the present invention has noticed that various attributes of the face affect the accuracy of face recognition. Therefore, the method combines face recognition and face attribute recognition, extracts the distilled depth face features and face attribute features by introducing the mutually antagonistic objective function, solves the problem that the similar inner distance of the same face is too large due to attribute reasons, and can simultaneously improve the accuracy of two tasks. In addition, the principles of the present invention are equally applicable to other application scenarios where relevant features within the input affect each other.
In order to clarify the technical solution of the present invention more clearly, the basic principle of the neural network is first explained. The data compression scheme of the present application is applicable to a variety of artificial neural networks, including Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), and Convolutional Neural Networks (CNNs). The following description will be made by taking CNN as an example.
CNN basic concept
CNNs achieve the most advanced performance in a wide range of, for example, vision-related tasks. The basic knowledge of CNN will be introduced based on the CNN image classification algorithm as follows.
as shown in fig. 1, a typical CNN consists of a series of layers that run in order.
The CNN neural network is composed of an input layer, an output layer and a plurality of hidden layers which are connected in series. The first layer of CNN reads an input value, such as an input image, and outputs a series of activation values. The lower layer reads the activation value generated by the previous layer and outputs a new activation value. The last classifier (classifier) outputs the probability of each class to which the input image may belong.
These layers can be broadly divided into weighted layers (e.g., convolutional layers, fully-connected layers, batch normalization layers, etc.) and unweighted layers (e.g., pooling layers, ReLU layers, Softmax layers, etc.). Here, the CONV layer (convolution layer) takes a series of feature maps as input, and is convolved with a convolution kernel to obtain an output activation value. The pooling layer is typically connected to the CONV layer for outputting a maximum or average value for each partition (sub area) in each feature map, thereby reducing the computational effort by sub-sampling while maintaining some degree of displacement, scale and deformation invariance. Multiple alternations between convolutional and pooling layers may be included in a CNN, thereby gradually reducing the spatial resolution and increasing the number of feature maps. A one-dimensional vector output comprising a plurality of eigenvalues may then be derived by applying a linear transformation on the input eigenvector, possibly connected to at least one fully connected layer.
In general, the operation of weighted layers can be represented as:
Y＝WX+b，
where W is the weight value, b is the offset, X is the input activation value, and Y is the output activation value.
The operation of the unweighted layer can be represented as:
Y＝f(X)，
Wherein f (X) is a non-linear function.
here, "weights" (weights) refer to parameters in the hidden layer, are values learned through the training process, and remain unchanged at inference; the activation value refers to a value, also referred to as a feature value, transferred between layers, starting from an input layer, and an output of each layer is obtained by an operation of the input value and a weight value. Unlike the weight values, the distribution of activation values varies dynamically according to the input data sample.
Before using CNN for reasoning (e.g., image classification), CNN needs to be trained first. Through the massive import of training data, parameters such as weights and offsets of each layer of the neural network model are determined. The training of CNN is mainly implemented on large servers. For embedded platforms, such as SoC platforms including FPGAs, focus on accelerating and implementing the inference computation process of CNN.
The training process of CNN, in the case where a network model has been determined, is a process of finding the weights and offsets required for the model so that the output y (x) of the network can fit all the training inputs x. To quantify this goal, it is often necessary to define an objective function (also referred to as a cost function, or loss function). To find a series of weights and biases that can be made as cost-effective as possible, a gradient descent algorithm is usually employed for this purpose.
although the continuous optimization of the objective function can greatly improve the reasoning accuracy of the neural network in an early stage, due to the inherent problem of class inner distance, the improvement effect of the classification and identification accuracy of the neural network by improving the objective function alone is less and less obvious.
The distance within a class (also referred to as an intra-class distance) is a distance between points corresponding to samples classified into the same class in a feature space in a classifier. For example, in face recognition, the activation value of the last fully-connected layer is input as a feature, the weight of the last fully-connected layer is normalized, and the normalized weight represents the center of the feature. Although the feature points extracted after different images of the same person are input into the trained face recognition neural network can be generally gathered around the feature center, the distribution distance of each feature point is still a certain distance away from the feature center due to the influence of various attributes of the face. For the class with less pictures, because the included angle between the feature center of the class and the feature center of the adjacent class is smaller, the feature space of the class is also smaller, and the probability of the class being classified by the classifier is also increased.
Obviously, the above problem of recognition accuracy caused by too large inter-class distance cannot be solved simply by improving the objective function. In view of this, the present invention provides a new neural network computing scheme, which can make the obtained target feature closer to the class feature center of the class by reducing the influence of other irrelevant features in the extracted target feature, thereby improving the classification accuracy of the neural network.
FIG. 2 shows a flow diagram of a neural network computational method, according to one embodiment of the invention.
In step S210, inputs to the neural network computation are obtained, the inputs comprising two or more classes of features. The above-described features may be related features, in other words, the above-described features have a correlation resulting from feature extraction of each other. In one embodiment, the input may be an image used for neural network inference calculations, such as an image containing a human face. The image comprises human face characteristics and human face attribute characteristics. Here, the face feature refers to a feature required for face recognition, such as a feature of a key point of five sense organs. The face attribute features refer to features required for attribute recognition such as gender, age, posture, expression and the like. Other relevant features may also be included in the image, such as a shot angle feature and a lighting condition feature.
In step S220, a first feature of the input for the first class is extracted based on a first class neural network. In one embodiment, the first class neural network may be a face recognition neural network and the first features may be facial features, for example, facial feature keypoint features.
In step S230, a second reference feature of the input for a second class is extracted based on the second class reference neural network. In one embodiment, the second category may relate to attributes of a human face. Therefore, the face attribute feature for reference can be extracted from the input image based on the face attribute recognition neural network for reference.
in step S240, the second reference feature is introduced into the first feature to remove the influence of the second feature in the first feature. Subsequently, in step S250, a neural network classification calculation for the first feature is completed based on the first feature superimposed with the second reference feature. Specifically, the first feature superimposed with the second reference feature may be fed into a classifier of the first class neural network to obtain a classification result.
in one embodiment, in step S240, the face feature vector extracted in step S220 may be superimposed with the face attribute reference feature vector extracted in step S230. The obtained result vector is a face feature vector with the influence of the face attribute removed to a certain extent. The resulting result vector may then be fed into a classifier of a face recognition neural network in step S250 to obtain a more accurate classification result.
based on the neural network computing method, the relevant reference feature vectors can be superposed before the extracted target feature vectors are sent to the final classifier of the neural network, so that the influence of relevant features on the positions of the target feature vectors is removed. The obtained result vector is closer to the class feature center, so that the accuracy of the classification task is improved.
in one embodiment, the relevant features of the first category may also be used in turn to improve the accuracy of the classification task of the second category. Fig. 3 shows the relevant steps of a preferred sub-scheme of the neural network computing method according to the present invention.
In step S310, a second feature of the input for a second class is extracted based on the second class neural network. In one embodiment, the second class neural network may be a facial attribute recognition neural network, and the extracted second feature may be a facial attribute feature such as age, expression, and the like, corresponding to the description of fig. 2.
In step S320, a first reference feature for a first class is extracted for an input based on the first class reference neural network. In one embodiment, the first category may be related to face recognition. Therefore, the face features for reference can be extracted from the input image based on the face recognition neural network for reference.
Subsequently, in step S330, the first reference feature is introduced into the second feature to remove the influence of the first feature in the second feature. In step S340, a neural network classification calculation for the second feature is completed based on the second feature superimposed with the first reference feature.
In one embodiment, in step S330, the face attribute feature vector extracted in step S310 may be superimposed with the face reference feature vector extracted in step S320. The obtained result vector is a face attribute feature vector with the influence of the face removed to a certain extent. Subsequently, the obtained result vector may be fed into the classifier of the face attribute recognition neural network in step S340 to obtain a more accurate classification result.
here, the second reference feature may be a second feature that does not include the first category feature information. Accordingly, the first reference feature is also a first feature that does not contain the second category feature information. For example, the face reference feature may be a face feature without the influence of the face attribute feature information, that is, a face feature that cannot identify any face attribute. The face attribute reference feature may be a face attribute feature from which the influence of the face feature is removed, that is, a face attribute feature from which any face feature cannot be recognized.
The reference features may be obtained using the reference neural network described above. The reference neural network may be a side branch network of the main neural network. In one embodiment, the second class of reference neural network is a side branch neural network of the second class of neural network, and/or the first class of reference neural network is a side branch neural network of the first class of neural network. The weights and biases of the side branch neural network are trained based on an objective function that competes against the first and second features. Preferably, the objective function of the countermeasure can be achieved by inserting a gradient inversion layer to remove the effect of other features. It should be understood that the countermeasure goals for removing the effects of other features may also be achieved by other means known to those skilled in the art.
It should be understood that steps S310-S340 of fig. 3 may be combined anywhere after the input acquisition step S210 shown in fig. 2, or performed in parallel or interactively with steps S220-S250, as long as the above steps are performed without affecting the removal of the reference features from each other. In addition, it should be understood that the present invention does not provide for the order of acquiring the characteristic information and the reference characteristic information, for example, step S230 may occur simultaneously with or before step S220, and step S320 may occur simultaneously with or before step S310.
The present invention also includes a neural network computing system corresponding to the neural network computing method shown in fig. 2 and 3. FIG. 4 shows a schematic diagram of a neural network computing system, in accordance with one embodiment of the present invention.
As shown in fig. 4, the neural network computing system 400 includes a first class neural network 410 and a second class reference neural network 420. The first class neural network 410 may be used to extract first features for a first class from the input, e.g., facial features from the input image. The second class reference neural network 420 may then be used to extract second reference features for a second class from the input, e.g. face attribute reference features from the same input image. The first class neural network 410 further comprises an overlap module 411, which introduces the second reference feature into the first feature to remove the influence of the second feature in the first feature, and the first class neural network 410 performs a neural network classification calculation for the first feature based on the first feature on which the second reference feature is overlapped. For example, before the extracted face feature vector is sent to the classifier, the face recognition neural network is added to the face attribute reference feature vector extracted from the face attribute recognition reference network to remove the influence of the face attribute on the face feature, and the obtained result vector is sent to the classifier of the face recognition neural network for classification, so that a more accurate classification result is ensured.
Fig. 5 shows an example of feature superposition in the feature space. As shown in the figure, the obtained result vector 3 of the obtained face feature vector 1 after the face attribute reference feature vector 2 is superimposed is closer to the class feature center 4 of the class in which the face is located, so that the face feature vector is less likely to be mistakenly classified into other classes in the subsequent classification process.
Accordingly, FIG. 6 shows a schematic diagram of a neural network computing system, in accordance with another embodiment of the present invention. In addition to the first category neural network 610 and the second category reference neural network 620 corresponding to those shown in fig. 4, a second category neural network 630 and a first category reference neural network 640 are included.
The second class neural network 630 is used to extract a second feature for a second class from the input, for example, the face attribute recognition neural network extracts a face attribute feature from the same input image. The first class reference neural network 640 may be used to extract a first reference feature for the first class from the input, for example, to extract a face recognition reference feature. Likewise, the second class neural network 630 further includes an overlap module 631, which introduces the first reference feature into the second feature to remove the influence of the first feature in the second feature, and the second class neural network 630 performs a neural network classification calculation for the second feature based on the second feature on which the first reference feature is overlapped.
In other words, although the principles of the present invention may be applied to the case where a single network overlaps a single reference network as shown in fig. 2 and 4, it is more applicable to the case where two backbone networks are included and two corresponding side branch reference networks are overlapped. In the present invention, the reference network may be a network associated with a backbone network, and the reference network may share most of the network structure with the backbone network, and only differs at the tail end. In one embodiment, the reference feature for removing the influence of the local backbone network feature in another backbone network can be obtained by performing a countermeasure function training on a side branch network (as a reference network) branched from the backbone network end.
In one embodiment, the backbone network and the reference network may share a single feature extraction network. For example, the first class neural network and the first class reference neural network may obtain the same first feature vector, which is then fed into the first class reference neural network as a side branch and removed from other feature influences by, for example, a countering objective function to obtain a first reference feature vector. Accordingly, the second class neural network and the second class reference neural network may obtain the same second feature vector, and then the second feature vector is fed into the second class reference neural network as a side branch and the influence of other features is removed by, for example, a countering objective function to obtain a second reference feature vector. The second reference feature vector is then superimposed with the previously obtained first feature vector to obtain a resultant first feature vector, which is fed into the classifier of the first class neural network to achieve a more accurate classification result. Accordingly, the first reference feature vector is then superimposed with the previously obtained second feature vector to obtain a resultant second feature vector, which is fed into the classifier of the second class neural network to achieve a more accurate classification result.
In other words, the present invention can achieve joint learning of the countermeasure target at a small cost. For the backbone network, only one vector superposition step needs to be added, and the side branch network can be simply realized by removing other associated features in the acquired backbone network features.
FIG. 7 shows a schematic diagram of a twin neural network, in accordance with one embodiment of the present invention. The dual neural network 700 in the figure jointly learns the method of two tasks of face recognition and face attribute recognition through the confrontation target. The basic structure of the network 700 is a double artificial neural network and cross information transfer between the neural networks. The trunk of the dual neural network is used as the trunk networks 710 and 730 for face recognition and face attribute recognition, respectively. The intersection between networks serves as a channel for the information transfer of two tasks, namely face recognition and face attribute recognition, namely reference networks 720 and 740. For the cross part of the network, supervised learning is carried out by a confrontation objective function, namely the information transmitted to the face recognition network by the face attribute recognition network is not the face attribute information directly acquired from the backbone network, but does not contain the face recognition information, namely the face attribute reference information of the face cannot be distinguished. Accordingly, the information transmitted to the face attribute recognition network by the face recognition network is not the face information directly acquired from the backbone network, but does not include the face attribute information, that is, the face reference information that cannot recognize the face attribute.
As shown, the reference networks 720 and 740 may share most of the network structure with the backbone network. For example, the reference network may directly acquire the features extracted by the backbone network and remove the influence of other features in the features by counteracting the objective function to acquire the reference features for being fed into another backbone network.
When the dual-network system is trained, the two main neural networks can respectively enable the classification target functions of face recognition and face attribute recognition to be used as supervision signals. For two pictures of the same face, because the difference of the face attributes causes the difference of depth features extracted from the main part of the face recognition network, under the ideal condition, the same features should be extracted from different pictures of the same face through the face recognition network, the difference between the features is caused by the face attribute difference to a great extent, and the mapping relation between the difference between the features and the face attribute features can be learned by introducing information from the face attribute recognition network, so that the converted features extracted from the face attribute recognition network can be superposed on the features extracted from the face recognition network. For different pictures with the same face attribute, the difference between the features extracted by the face attribute network is largely because the two pictures are not the same person, so that the mapping relation between the difference between the features and the face features can be learned by introducing information from the face recognition network. Thus, the backbone network comprises respective overlay modules 711 and 731 for overlaying the features obtained from the backbone network with reference features from another side branch network to try to remove the influence of other relevant classes on the current class feature determination.
Therefore, through the cross counterwork learning of the two networks, the class internal distance of the features extracted by the two networks is smaller, and information irrelevant to the main task is less, so that the accuracy of face recognition and face attribute recognition is improved.
The above describes in connection with fig. 2-7 the case where the input data has two related categories, for example a first category and a second category. It should be understood that the neural network computational scheme of the present invention is also applicable to situations having a plurality of second class features. In other words, the second category includes a plurality of other categories of the features of the two or more categories excluding the first category. For example, in the input face image, in addition to face recognition and face attribute features, features such as illumination and shooting angle are included. When the face features are regarded as the first category, the face attribute features, the illumination features, and the shooting angle features may be regarded as a plurality of second categories.
Therefore, the influence of the plurality of second features on the first feature may be removed by introducing a plurality of reference features through a plurality of second reference neural networks, or may be removed by a single reference neural network incorporating the plurality of second features, which is not limited in the present invention. With the introduction of multiple reference features, the principles of the present invention are also applicable to three-network or even more neural network confrontational target joint learning schemes.
Thus, in one embodiment, extracting a second reference feature of the input for a second class based on a second class reference neural network may comprise: extracting a plurality of second reference features based on the plurality of second class reference neural networks; and introducing the second reference feature into the first features to remove the effect of the second one of the first features comprises: a plurality of second reference features are respectively introduced into the first features to remove the effect of each of the second features in the first features.
In another embodiment, extracting a second reference feature of the input for a second class based on a second class reference neural network may include: extracting a second reference feature fusing the other multiple class features based on a single second class reference neural network fusing the other multiple classes.
accordingly, the second class of reference neural network may be a plurality of second class of reference neural networks each based on one of the other plurality of classes, or may be a single second class of reference neural network based on the other plurality of classes.
Fig. 8 is a schematic structural diagram of a data processing computing device that can be used to implement the neural network computing method according to an embodiment of the present invention.
Referring to fig. 8, computing device 800 includes memory 810 and processor 820.
The processor 820 may be a multi-core processor or may include multiple processors. In some embodiments, processor 820 may include a general-purpose host processor and one or more special coprocessors such as a Graphics Processor (GPU), a Digital Signal Processor (DSP), or the like. In some embodiments, processor 820 may be implemented using custom circuitry, such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA).
The memory 810 may include various types of storage units, such as system memory, Read Only Memory (ROM), and permanent storage. Wherein the ROM may store static data or instructions that are needed by the processor 820 or a second module of the computer. The persistent storage device may be a read-write storage device. The persistent storage may be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the persistent storage device employs a mass storage device (e.g., magnetic or optical disk, flash memory) as the persistent storage device. In other embodiments, the permanent storage may be a removable storage device (e.g., floppy disk, optical drive). The system memory may be a read-write memory device or a volatile read-write memory device, such as a dynamic random access memory. The system memory may store instructions and data that some or all of the processors require at runtime. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic and/or optical disks, may also be employed. In some embodiments, memory 1010 may include a removable storage device that is readable and/or writable, such as a Compact Disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, min SD card, Micro-SD card, etc.), a magnetic floppy disc, or the like. Computer-readable storage media do not contain carrier waves or transitory electronic signals transmitted by wireless or wired means.
The memory 810 has stored thereon processable code, which, when processed by the processor 820, causes the processor 820 to perform the neural network computing methods described above.
The neural network calculation scheme and the dual neural network implementation according to the present invention have been described in detail above with reference to the accompanying drawings.
Furthermore, the method according to the invention may also be implemented as a computer program or computer program product comprising computer program code instructions for carrying out the above-mentioned steps defined in the above-mentioned method of the invention.
Alternatively, the invention may also be embodied as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having stored thereon executable code (or a computer program, or computer instruction code) which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the above-described method according to the invention.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.
In addition, "first" and "second" are used herein only to distinguish between different juxtaposed categories, and do not imply a sequential or significant relationship between the two. For example, in the example of the present invention, the first class neural network may also be a face attribute recognition neural network, and the second class neural network may be a face recognition neural network.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (21)
1. a neural network computing method, comprising:
Obtaining input for neural network computations, the input comprising features of two or more categories;
Extracting a first feature of the input for the first class based on a first class neural network;
extracting a second reference feature of the input for a second class based on a second class reference neural network;
introducing the second reference feature into the first feature to remove the effect of a second one of the first features; and
Completing a neural network classification calculation for the first feature based on the first feature superimposed with the second reference feature.
2. The method of claim 1, further comprising:
extracting a second feature of the input for a second category based on a second category neural network;
Extracting a first reference feature of the input for the first class based on a first class reference neural network;
Introducing the first reference feature into the second feature to remove the effect of the first one of the second features; and
Completing a neural network classification calculation for the second feature based on the second feature superimposed with the first reference feature.
3. The method according to claim 1 or 2, wherein the second reference feature is a second feature not comprising feature information of the first category, and/or,
The first reference feature is a first feature that does not include second category feature information.
4. The method of claim 3, wherein the second class of reference neural network is a side branch neural network of the second class of neural network, and/or,
the first class reference neural network is a side branch neural network of the first class neural network.
5. the method of claim 4, wherein the weights and biases of the side branch neural network are trained based on an objective function that competes against the first and second features.
6. The method of claim 5, wherein the objective function of the countermeasure is achieved by inserting gradient inversion layers.
7. The method of claim 1 or 2, wherein the input is an image, the first class of neural network is a face recognition neural network, the first feature is a face feature, the second class of neural network is a face attribute recognition neural network, and the second feature is a face attribute feature.
8. The method of claim 1, wherein the second category comprises a plurality of other categories of the two or more categories of features excluding the first category.
9. The method of claim 8, wherein extracting a second reference feature of the input for a second class based on a second class reference neural network comprises:
Extracting a plurality of second reference features based on the plurality of second class reference neural networks; and is
Introducing the second reference feature into the first feature to remove the effect of the second one of the first features comprises:
a plurality of second reference features are respectively introduced into the first features to remove the effect of each of the second features in the first features.
10. The method of claim 8, wherein extracting a second reference feature of the input for a second class based on a second class reference neural network comprises:
Extracting a second reference feature fusing the other multiple class features based on a single second class reference neural network fusing the other multiple classes.
11. The method of claim 1, wherein completing neural network computations for the first feature based on the first feature with the second reference feature superimposed comprises:
and sending the first feature superposed with the second reference feature into a classifier of the first-class neural network to obtain a classification result.
12. A neural network computing system, comprising:
A first class neural network for extracting a first feature for a first class from an input;
A second class reference neural network for extracting a second reference feature for a second class from the input,
Wherein the first class neural network further comprises an overlap-add module which introduces the second reference feature into the first feature to remove an influence of a second feature in the first feature, and the first class neural network completes a neural network classification calculation for the first feature based on the first feature on which the second reference feature is overlapped.
13. the system of claim 12, further comprising:
A second class neural network for extracting second features for a second class from the input;
A first class reference neural network for extracting a first reference feature for a first class from the input,
Wherein the second class neural network further comprises an overlap-add module which introduces the first reference feature into the second feature to remove the influence of the first feature in the second feature, and the second class neural network completes the neural network classification calculation for the second feature based on the second feature on which the first reference feature is overlapped.
14. The system of claim 13, wherein the first class of reference neural network is a side branch neural network of the first class of neural network, and
The second class of reference neural networks are side branch neural networks of the second class of neural networks.
15. the system of claim 14, wherein the weights and biases of the side branch neural network are trained based on an objective function that competes against the first and second features, and the reference features derived by the side branch neural network are features that do not contain other class feature information.
16. The system of claim 13, wherein the input is an image, the first class neural network is a face recognition neural network, the first feature is a facial feature, and,
The second class neural network is a face attribute recognition neural network, and the second feature is a face attribute feature.
17. The system of claim 13, wherein the second category comprises a plurality of other of the two or more categories of features excluding the first category, and wherein:
The second class of reference neural networks is a plurality of second class of reference neural networks each based on one of the other plurality of classes, and/or
The second class of reference neural network is a single second class of reference neural network based on other multiple classes.
18. a dual neural network for joint computation, comprising:
a face recognition neural network as a backbone neural network for extracting face features from the input image;
a face attribute recognition neural network as another backbone neural network for extracting a face attribute feature from the input image;
a face recognition reference neural network serving as a side branch network of the face recognition neural network extracts face reference features which do not contain face attribute information from the input image;
A face attribute recognition reference neural network as a side branch network of the face attribute recognition neural network, extracting a face attribute reference feature not containing face information from the input image,
The face recognition neural network comprises an overlapping module, the overlapping module introduces the face attribute reference features into the face features to remove the influence of the face attribute features in the face features, and the face recognition neural network completes the neural network calculation aiming at the face features based on the face features overlapped with the face attribute reference features.
19. The dual neural network of claim 18, the side branch neural network and the trunk neural network share most of the preceding network structure, and the reference features are found based on weights and bias parameters trained on an objective function that competes against face and face attribute features.
20. A computing device, comprising:
A processor; and
A memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the method of any one of claims 1-11.
21. A non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-11.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810550516.1A CN110555340B (en) | 2018-05-31 | 2018-05-31 | Neural network computing method and system and corresponding dual neural network implementation |
| PCT/IB2019/000603 WO2019229524A2 (en) | 2018-05-31 | 2019-05-30 | Neural network calculation method and system, and corresponding dual neural network implementation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810550516.1A CN110555340B (en) | 2018-05-31 | 2018-05-31 | Neural network computing method and system and corresponding dual neural network implementation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110555340A true CN110555340A (en) | 2019-12-10 |
| CN110555340B CN110555340B (en) | 2022-10-18 |
Family
ID=68697424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810550516.1A Active CN110555340B (en) | 2018-05-31 | 2018-05-31 | Neural network computing method and system and corresponding dual neural network implementation |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110555340B (en) |
| WO (1) | WO2019229524A2 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111723691A (en) * | 2020-06-03 | 2020-09-29 | 北京的卢深视科技有限公司 | Three-dimensional face recognition method and device, electronic equipment and storage medium |
| CN112257526A (en) * | 2020-10-10 | 2021-01-22 | 中国科学院深圳先进技术研究院 | Action identification method based on feature interactive learning and terminal equipment |
| CN112395971A (en) * | 2020-11-16 | 2021-02-23 | 公安部第三研究所 | StarGAN-based face test image generation method under different quantitative illumination and angle conditions, application and storage medium |
| CN115034312A (en) * | 2022-06-14 | 2022-09-09 | 燕山大学 | Fault diagnosis method for dual neural network model satellite power supply system |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112766215A (en) * | 2021-01-29 | 2021-05-07 | 北京字跳网络技术有限公司 | Face fusion method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20160122452A (en) * | 2015-04-14 | 2016-10-24 | (주)한국플랫폼서비스기술 | Deep learnig framework and image recognition method for content-based visual image recognition |
| CN107145857A (en) * | 2017-04-29 | 2017-09-08 | 深圳市深网视界科技有限公司 | Face character recognition methods, device and method for establishing model |
| CN107766850A (en) * | 2017-11-30 | 2018-03-06 | 电子科技大学 | Based on the face identification method for combining face character information |
| CN107895160A (en) * | 2017-12-21 | 2018-04-10 | 曙光信息产业(北京)有限公司 | Human face detection and tracing device and method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2955662B1 (en) * | 2003-07-18 | 2018-04-04 | Canon Kabushiki Kaisha | Image processing device, imaging device, image processing method |
| US9471886B2 (en) * | 2013-10-29 | 2016-10-18 | Raytheon Bbn Technologies Corp. | Class discriminative feature transformation |
| US10346740B2 (en) * | 2016-06-01 | 2019-07-09 | Kla-Tencor Corp. | Systems and methods incorporating a neural network and a forward physical model for semiconductor applications |
| CN107403200A (en) * | 2017-08-10 | 2017-11-28 | 北京亚鸿世纪科技发展有限公司 | Improve the multiple imperfect picture sorting technique of image segmentation algorithm combination deep learning |
-
2018
- 2018-05-31 CN CN201810550516.1A patent/CN110555340B/en active Active
-
2019
- 2019-05-30 WO PCT/IB2019/000603 patent/WO2019229524A2/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20160122452A (en) * | 2015-04-14 | 2016-10-24 | (주)한국플랫폼서비스기술 | Deep learnig framework and image recognition method for content-based visual image recognition |
| CN107145857A (en) * | 2017-04-29 | 2017-09-08 | 深圳市深网视界科技有限公司 | Face character recognition methods, device and method for establishing model |
| CN107766850A (en) * | 2017-11-30 | 2018-03-06 | 电子科技大学 | Based on the face identification method for combining face character information |
| CN107895160A (en) * | 2017-12-21 | 2018-04-10 | 曙光信息产业(北京)有限公司 | Human face detection and tracing device and method |
Non-Patent Citations (2)
| Title |
|---|
| GUOSHENG HU ET AL.: "Attribute-enhanced face recognition with neural tensor fusion networks", 《2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION（ICCV）》 * |
| 邵蔚元等: "多任务学习及卷积神经网络在人脸识别中的应用", 《计算机工程与应用》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111723691A (en) * | 2020-06-03 | 2020-09-29 | 北京的卢深视科技有限公司 | Three-dimensional face recognition method and device, electronic equipment and storage medium |
| CN111723691B (en) * | 2020-06-03 | 2023-10-17 | 合肥的卢深视科技有限公司 | Three-dimensional face recognition method and device, electronic equipment and storage medium |
| CN112257526A (en) * | 2020-10-10 | 2021-01-22 | 中国科学院深圳先进技术研究院 | Action identification method based on feature interactive learning and terminal equipment |
| CN112257526B (en) * | 2020-10-10 | 2023-06-20 | 中国科学院深圳先进技术研究院 | Action recognition method based on feature interactive learning and terminal equipment |
| CN112395971A (en) * | 2020-11-16 | 2021-02-23 | 公安部第三研究所 | StarGAN-based face test image generation method under different quantitative illumination and angle conditions, application and storage medium |
| CN115034312A (en) * | 2022-06-14 | 2022-09-09 | 燕山大学 | Fault diagnosis method for dual neural network model satellite power supply system |
| CN115034312B (en) * | 2022-06-14 | 2023-01-06 | 燕山大学 | Fault diagnosis method for dual neural network model satellite power system |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019229524A2 (en) | 2019-12-05 |
| CN110555340B (en) | 2022-10-18 |
| WO2019229524A3 (en) | 2020-05-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110555340B (en) | Neural network computing method and system and corresponding dual neural network implementation | |
| Rocco et al. | Efficient neighbourhood consensus networks via submanifold sparse convolutions | |
| Lee et al. | Salient region-based online object tracking | |
| CN111814794B (en) | Text detection method and device, electronic equipment and storage medium | |
| Xia et al. | Loop closure detection for visual SLAM using PCANet features | |
| EP2385483A1 (en) | Recognition and pose determination of 3D objects in 3D scenes using geometric point pair descriptors and the generalized Hough Transform | |
| CN110175615B (en) | Model training method, domain-adaptive visual position identification method and device | |
| CN114359851A (en) | Unmanned target detection method, device, equipment and medium | |
| Ansari | A review on SIFT and SURF for underwater image feature detection and matching | |
| US20210272295A1 (en) | Analysing Objects in a Set of Frames | |
| CN112115805A (en) | Pedestrian re-identification method and system with bimodal hard-excavation ternary-center loss | |
| CN111753119A (en) | Image searching method and device, electronic equipment and storage medium | |
| Mao et al. | 3dg-stfm: 3d geometric guided student-teacher feature matching | |
| CN116912796A (en) | Novel dynamic cascade YOLOv 8-based automatic driving target identification method and device | |
| CN115280373A (en) | Managing occlusions in twin network tracking using structured dropping | |
| CN110458234B (en) | Vehicle searching method with map based on deep learning | |
| Batool et al. | Ielmnet: An application for traffic sign recognition using cnn and elm | |
| CN115984671A (en) | Model online updating method and device, electronic equipment and readable storage medium | |
| Melo et al. | Learning to detect good keypoints to match non-rigid objects in rgb images | |
| Munoz et al. | Improving Place Recognition Using Dynamic Object Detection | |
| Yuanhong et al. | Infrared-visible image patches matching via convolutional neural networks | |
| CN111695380B (en) | Target detection method and device | |
| He et al. | SonarNet: Hybrid CNN-Transformer-HOG Framework and Multifeature Fusion Mechanism for Forward-Looking Sonar Image Segmentation | |
| Da Silva et al. | Visual Place Recognition Under a High-Dimensional Subspace Clustering Perspective | |
| Liu et al. | Adaptive Real-Time Loop Closure Detection Based on Image Feature Concatenation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200903 Address after: Unit 01-19, 10 / F, 101, 6 / F, building 5, yard 5, Anding Road, Chaoyang District, Beijing 100029 Applicant after: Xilinx Electronic Technology (Beijing) Co.,Ltd. Address before: 100083, 17 floor, four building four, 1 Wang Zhuang Road, Haidian District, Beijing. Applicant before: BEIJING DEEPHI INTELLIGENT TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |