WO2023072094A1

WO2023072094A1 - Visualization and quantitative analysis method and system for expression capability of layer feature in neural network

Info

Publication number: WO2023072094A1
Application number: PCT/CN2022/127435
Authority: WO
Inventors: 张拳石
Original assignee: 上海交通大学
Priority date: 2021-10-25
Filing date: 2022-10-25
Publication date: 2023-05-04
Also published as: CN113963185A

Abstract

The present application relates to the technical field of machine learning. Disclosed are a visualization and quantitative analysis method and system for an expression capability of a layer feature in a neural network. The automatic visualization and quantitative analysis of an expression capability of a layer feature in a neural network can be realized under an unsupervised condition. The method comprises: providing a neural network to be analyzed, wherein the neural network is a deep neural network, which is pre-trained on a certain data set; providing a group of input samples, inputting the samples into the neural network, and extracting sample-wise features and region-wise features corresponding to the samples; respectively performing dimension reduction on the sample-wise features and the region-wise features, so as to obtain visualization results in a low-dimensional space; and quantitatively analyzing the number and quality of knowledge points in the features on the basis of the visualization result of the region-wise features.

Description

A method and system for visualizing and quantitatively analyzing the expressive ability of middle-level features in neural networks

technical field

This application relates to the technical field of machine learning, and in particular to a method and system for visualizing and quantitatively analyzing the expressive ability of middle-level features of neural networks.

Background technique

At present, deep neural networks have demonstrated powerful performance in various fields, but the black-box nature of neural networks makes it difficult for people to understand their internal behavior. Among the existing technologies, the visualization method is the most widely used method in the field of artificial intelligence interpretability, but it is impossible to use the existing visualization method to quantitatively analyze the expressive ability of the layer features in the neural network.

Therefore, combining the visual interpretation of neural networks with the quantitative analysis of the expressive power of mid-level features is an urgent problem to be solved in the field of artificial intelligence interpretability.

Contents of the invention

The purpose of the present invention is to provide a method and system for visualizing and quantitatively analyzing the expressive ability of middle-level features of neural networks, which can automatically visualize and quantitatively analyze the expressive ability of middle-level features of neural networks under unsupervised conditions.

This application discloses a method for visualizing and quantitatively analyzing the expressive ability of the middle layer of the neural network, including the following steps

(1) Select the feature interpretation object:

Select the model to be analyzed, wherein the model includes a middle-level expression, including: a neural network, a hierarchical graph model;

(2) Extract neural network features:

Provide a set of input samples, input these samples into the above-mentioned neural network, and extract the features of these samples, wherein the features include: sample-wise feature and regional feature;

(3) Feature dimensionality reduction to obtain visualization results:

First, the dimensionality reduction of the sample-level features is performed to obtain the visualization results of the sample-level features in the low-dimensional space; secondly, based on the low-dimensional representation of the sample-level features and the regional-level features, the dimensionality reduction of the regional-level features is performed to obtain the regional-level features Visualization results in low-dimensional space;

(4) Quantitative analysis of features based on visualization results:

Based on the visualization results, the quantity and quality of knowledge points in the features are quantitatively analyzed.

In a preferred example, said step (1) further includes the following steps:

Based on a certain data set, train a neural network as the neural network to be analyzed. Optionally, the neural network is a classification neural network.

In a preferred example, said step (2) further includes the following sub-steps:

(a) Extraction of sample-level features: Input a given set of samples into the neural network to be analyzed, and for each sample, extract the output features of an intermediate layer of the neural network, so as to obtain the corresponding Sample-level features, the sample-level features corresponding to the set of input samples can be obtained.

(b) Extraction of region-level features: Input a given set of input samples into the neural network to be analyzed, and for each sample, extract the output features of a certain convolutional layer of the neural network to obtain each input sample The corresponding feature map (feature map), where the high-dimensional vector corresponding to each position of the feature map is the region-level feature of the sample in this region. When the height and width of this feature map are H and W respectively, and there are K channels in total, then this feature map contains HW region-level features, where each region-level feature is a K-dimensional vector.

In a preferred example, said step (3) further includes the following sub-steps:

(a) Dimensionality reduction of sample-level features and visualization in low-dimensional space;

(b) Dimensionality reduction is performed on region-level features and visualized in a low-dimensional space.

In the above sub-step (a), for each sample x corresponding to the sample level features

through a projection matrix

Map it into a low-dimensional space to obtain a low-dimensional representation of sample-level features

And, optimize M so that the low-dimensional representation should satisfy that the closeness of the low-dimensional representation g to each category should be consistent with the closeness of the sample x to each category as much as possible.

Optionally, in the calculation of the "closeness between low-dimensional representation and each category", first use the radial distribution (radial distribution) to model the distribution of the low-dimensional representation g of the sample-level feature in the low-dimensional space, and then based on this Computes how close a low-dimensional representation is to each class.

Based on the radial distribution, the probability density function of g in low-dimensional space can be calculated by the following formula.

Among them, y∈{1,2,…,C} represents different categories in the classification task; π _y represents the prior probability of the yth class; l _g =‖g‖ represents the L2 norm of g, which is called the strength of g (strength); o _g = g/l _g represents the orientation of g; μ _y represents the mean direction of the yth class; κ(·) is a monotonically increasing function. p(l _g |y) indicates the prior probability of l _g on category y, p _vMF (o _g |μ _y ,κ(l _g )) indicates that the average direction is μ _y , and the concentration parameter is κ( l _g ) vMF distribution (von Mises-Fisher distribution).

Optionally, κ(·) can be any monotonically increasing function, more preferably, the κ(·) function can be generated by the following method: Given a non-negative constant κ _m and dimension d, from the average direction for

N samples are obtained by sampling from the vMF distribution with aggregation parameter κ=κ _m

Scale these samples to a length of l without changing the direction, where l is an arbitrary non-negative number, and record the scaled samples as

Sample N samples of Gaussian noise from a standard normal distribution

Add the sample obtained by the aforementioned scaling to the Gaussian noise sample correspondingly, and obtain

Define κ(l) as

in

The range of N is 1000-100000, and in one embodiment, N is taken as 10000.

Based on the radial distribution above, and assuming that the prior probability of l _g is independent of category y, then the closeness Q _M (y|x) between the low-dimensional representation g and the yth category can be calculated by the following formula.

Optionally, the "closeness between the sample and each category" is calculated as the output probability of the sample, that is, the closeness P(y|x) between the sample x and the yth category is the output probability corresponding to the yth category in the neural network output value.

Optionally, the optimization of the projection matrix M further includes the calculation results of the closeness Q _M (y|x) between the low-dimensional representation g and each category, and the closeness P(y|x) between the sample x and each category , optimize the projection matrix M so that the KL divergence (Kullback–Leibler divergence) between P(y|x) and Q _M (y|x) is minimized.

Optionally, in the process of obtaining the low-dimensional representation of sample-level features, alternately optimize the projection matrix M and the parameters {π,μ}={π _y ,μ _y } _y∈Y in the radial distribution. Among them, when optimizing the projection matrix M, the parameters {π, μ} in the radial distribution are fixed, and M is updated to minimize the value of KL[P(Y|X)‖Q _M (Y|X)]; the optimized path When the parameters in the distribution are {π, μ}, the projection matrix M is fixed, and {π, μ} is updated, so that the likelihood ∏ _g p(g)=∏ _g ∑ _y′ π _y′ ·p _vMF (o _g | μ _y′ ,κ(l _g )) has the largest value.

In the above sub-step (b), for each sample x's HW region-level features

through a projection matrix

Map them into a low-dimensional space to obtain a low-dimensional representation of HW region-level features

And, optimize Λ so that the low-dimensional representation should satisfy that the similarity between samples inferred based on the low-dimensional representation h={h ⁽¹⁾ ,h ⁽²⁾ ,…,h ^(HW) } is the same as that inferred based on the network output The similarity between samples obtained should be as consistent as possible. Furthermore, the low-dimensional representation of region-level features needs to be aligned with the low-dimensional feature representation of sample-level features.

Optionally, in the calculation of "similarity between samples inferred based on low-dimensional representation", based on the bag-of-words model (the bag-of-words model), the similarity between samples is split into regions corresponding to low-dimensional The weighted product of the similarity between representations; and based on the vMF distribution, the similarity between the corresponding low-dimensional representations of each region is quantified.

Optionally, in the above description, let _x1 and _x2 be any two samples, and the low-dimensional representations of the corresponding region-level features are respectively

and

Based on the bag-of-words model, the similarity Q _Λ (x ₂ |x ₁ ) between x ₁ and x ₂ is split into the weighted product of the similarity between the corresponding low-dimensional representations of each region, as shown below.

in,

Indicates the importance of the rth regional feature in the sample x ₂ for classification. In a preferred example,

is a non-negative number. and,

It is further quantified into the following form.

in,

for

in the mean direction of

The aggregation parameters are

The probability density in the distribution of vMF; as described in claim 6, κ(·) is a monotonically increasing function.

Optionally, in the calculation of "similarity between samples inferred based on network output", let _x1 and _x2 be any two samples,

are the network output probabilities corresponding to the two samples, and the similarity P(x ₂ |x ₁ ) between samples inferred based on the network output can be further calculated as the following form.

in,

cos(·,·) represents the cosine similarity between two vectors, and κ _p is a non-negative constant.

Optionally, making "the similarity between samples inferred based on low-dimensional representations and the similarity between samples inferred based on network output as consistent as possible" is equivalent to minimizing the following loss function.

Wherein, the calculations of P(x ₂ |x ₁ ) and Q _Λ (x ₂ |x ₁ ) are as shown in claim 16 and claim 17 respectively.

Optionally, aligning the low-dimensional representation of region-level features with the low-dimensional representation of sample-level features is equivalent to optimizing the following loss function.

Among them, MI(·;·) represents mutual information; g represents the low-dimensional representation of the sample-level features of sample x; h ^(r) represents the low-dimensional representation of the r-th region-level features of sample x, w ^{( r)} indicates the importance of the r-th region-level feature of sample x for classification.

Optionally, the optimization of the projection matrix Λ includes the following steps: respectively calculating the above two loss functions

and

Then calculate the total loss function

Optimizing Λ based on the total loss function such that the total loss function

minimize. Wherein, α is a positive constant, and in a preferred example, the value range of α is 0.01 to 100. In one embodiment, the value of α is taken as 0.1.

In a preferred example, said step (4) further includes the following steps:

(a) Quantify knowledge points in region-level features;

(b) Further quantify reliable knowledge points and the proportion of reliable knowledge points.

Optionally, a knowledge point is defined as a set of region-level features such that the following formula is greater than a certain threshold.

Among them, h ^(r) is the low-dimensional representation of the r-th region-level feature corresponding to a certain sample x. Therefore, knowledge points represent region-level features such that max _c p(y=c|h ^(r) )>τ, that is, the set {h ^(r) : max _c p(y=c|h ^(r) )>τ} The region-level features included in , where τ is a positive constant, and in a preferred example, the value range of τ is 0.3-0.8. In one embodiment, the value of τ is 0.4.

Optionally, it is further possible to quantify reliable knowledge points and unreliable knowledge points in the knowledge points, so as to quantify the proportion of reliable knowledge points to the total knowledge points. Among them, reliable knowledge points are the set of knowledge points that further satisfy the following formula.

Among them, h ^(r) is the low-dimensional representation of the r-th region-level feature corresponding to a certain sample x, and c ^truth represents the true category label of the sample. That is, reliable knowledge points are region-level features included in the set {h ^(r) :c ^truth =argmax _c p(y=c|h ^(r) )}.

Further, the ratio of reliable knowledge points to total knowledge points measures the quality of knowledge points, which can be calculated by the following formula.

The second aspect of the present invention provides a visualization and quantitative analysis system for the expressive ability of the middle layer of the neural network, which is characterized in that it includes:

(1) The input module is configured as a pre-trained classification neural network and contains input samples of all possible categories;

(2) a feature extraction module configured to extract sample-level features and region-level features of the input samples;

(3) a visualization module configured to, based on the extracted sample-level features and region-level features, reduce its dimension to obtain a low-dimensional representation, and visualize the low-dimensional representation in the low-dimensional space;

(3) The quantitative analysis module is configured to quantitatively analyze the quantity and quality of knowledge points in the feature based on the visualization result of the feature at the region level.

It should be understood that within the scope of the present invention, the above-mentioned technical features of the present invention and the technical features specifically described in the following (such as embodiments) can be combined with each other to form new or preferred technical solutions. Due to space limitations, we will not repeat them here.

Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings.

Description of drawings

Fig. 1 is a schematic flow chart of a method for visualization and quantitative analysis of layer feature expression capabilities in a neural network according to a first embodiment of the present invention;

Fig. 2 is a schematic diagram of visualization of sample-level features in low-dimensional space obtained according to the present invention;

Fig. 3 is a schematic diagram of visualization of region-level features obtained according to the present invention in low-dimensional space at different training stages of the neural network;

Fig. 4 is a schematic diagram of visualization of region-level features obtained according to the present invention in low-dimensional space at different stages of forward propagation of the neural network;

Fig. 5 is according to the quantification to knowledge point quantity and quality in the present invention, obtains the knowledge point and reliable knowledge point in the different inter-layer features of neural network, with the change graph of different training stages of neural network;

Fig. 6 is the quantification of knowledge points according to the present invention, and the visualized diagram of knowledge points in different inter-layer features of the neural network obtained;

Fig. 7 is a schematic diagram of the system structure of the visualization and quantitative analysis of the feature expression ability of the middle layer of the neural network according to the second embodiment of the present invention.

Detailed ways

After careful and in-depth research, the present inventor has first developed a method and system for visualizing and quantitatively analyzing the expressive ability of middle-level features of neural networks. Through this method and system, the visual interpretation of the neural network is closely linked with the quantitative analysis of the feature expression ability of the middle layer of the neural network; through visualization, the emergence process of the feature expression ability of the middle layer of the neural network can be clearly displayed over time and space; this kind of The method can quantitatively analyze the quantity and quality of the knowledge points in the middle layer of the neural network, and then analyze the reliability of the model to be explained; based on this method and system, it can be used for many existing deep learning algorithms, such as adversarial attacks and knowledge distillation. An interpretive framework from a new perspective.

Based on this method and system, it can provide a new interpretation framework for many existing deep learning algorithms, such as adversarial attacks and knowledge distillation.

general method

Typically, the present invention includes the following steps:

(1) Provide a neural network to be analyzed, which is a pre-trained deep neural network on a certain data set;

(2) Provide a set of input samples, input these samples into the above neural network, and extract the sample-wise features and regional features corresponding to these samples;

(3) Dimensionality reduction is performed on sample-level features and region-level features to obtain visualization results in low-dimensional space;

(4) Quantitatively analyze the quantity and quality of knowledge points in features based on the visualization results of region-level features.

The main advantages of the present invention are:

(1) A method and system for visualization and quantitative analysis of the feature expression ability of the middle layer of the neural network;

(2) Through this method and system, the visual interpretation of the neural network is closely linked with the quantitative analysis of the expressive ability of the layer features in the neural network;

(3) Through visualization, it is possible to clearly show the emergence process of the feature expression ability of the middle layer of the neural network over time and space;

(4) This method can quantitatively analyze the quantity and quality of knowledge points in the middle layer of the neural network, and then analyze the reliability of the model to be explained;

(5) Based on this method and system, it can provide a new interpretation framework for many existing deep learning algorithms, such as adversarial attacks and knowledge distillation.

Example

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manner of the present application will be further described in detail below in conjunction with the accompanying drawings.

The first embodiment of the present invention designs a kind of visualization and quantitative analysis method to the feature expression ability of neural network middle layer, and its flow process is as shown in Figure 1, and this method comprises the following steps:

In step 101: based on a certain data set, train a neural network as the neural network to be analyzed. Optionally, the neural network is a classification neural network.

Afterwards, enter step 102, this step can be further divided into following two sub-steps:

Afterwards, enter step 103, this step can be further divided into following two sub-steps:

through a projection matrix

Sample N samples of Gaussian noise from a standard normal distribution

Define κ(l) as

in

The range of N is 1000-100000, and in one embodiment, N is taken as 10000.

As shown in Figure 2, in one embodiment of the present invention, a VGG-16 network pre-trained on the Tiny ImageNet image classification data set is given, wherein only the steel arch is used in the neural network during training The ten categories of bridge, school bus, sports car, tabby cat, desk, golden retriever, tailed frog, iPod, lifeboat, orange are extracted from the second-to-last fully connected layer of the VGG-16 network as sample-level features, using The aforementioned method performs dimensionality reduction to a low-dimensional representation scatter diagram obtained in a three-dimensional space. In the figure, different colors represent different categories shown in the legend, and the arrows corresponding to the colors of each category indicate the average direction of each category.

In the above sub-step (b), for each sample x's HW region-level features

through a projection matrix

and

in,

is a non-negative number. and,

It is further quantified into the following form.

in,

for

in the mean direction of

The aggregation parameters are

in,

Optionally, making "the similarity between samples inferred based on the low-dimensional representation and the similarity between samples inferred based on the network output as consistent as possible" is equivalent to minimizing the following loss function.

and

Then calculate the total loss function

As shown in Figure 3, given the same VGG-16 network pre-trained on the Tiny ImageNet dataset as mentioned above, the output features of the conv_53 layer are extracted as region-level features, and the dimensionality reduction to three dimensions is performed using the aforementioned method to obtain a low-dimensional representation. , the distribution in three-dimensional space. Among them, the same as above, the scatter points of different colors represent the low-dimensional representations of the region-level features corresponding to different types of samples, and the ellipsoids in the figure represent the approximate distribution of the low-dimensional representations of the region-level features corresponding to different types of samples.

As shown in Figure 4, given a VGG-16 network pre-trained on the Tiny ImageNet dataset as mentioned above, the output features of conv_12, conv_22, conv_33, conv_43, conv_53 layers are extracted as region-level features, and the aforementioned methods are used respectively Perform dimensionality reduction to three-dimensional space to obtain low-dimensional representation. Figure 4 shows the distribution of low-dimensional representations of regional-level features corresponding to different samples as the number of layers of forward propagation increases, where the vertical upward arrow represents the correct average direction of the sample, and the scatter points of different colors represent different The distribution of low-dimensional representations of region-level features corresponding to layers in three-dimensional space.

Afterwards, enter step 104, this step can be further divided into following sub-steps:

(a) Quantify knowledge points in region-level features;

As shown in Figure 5, given a VGG-16 network pre-trained on the Tiny ImageNet data set as mentioned above, the number of all knowledge points in the corresponding region-level features of the conv_33, conv_43, and conv_53 layers are calculated using the aforementioned method, And the number of reliable knowledge points. Figure 5 shows the change curve of the total amount of knowledge points and the number of reliable knowledge points at different layers of the neural network with the number of neural network training iterations.

As shown in Figure 6, given the same VGG-16 network pre-trained on the Tiny ImageNet dataset as mentioned above, all knowledge points in the corresponding region-level features of the conv_33, conv_43, and conv_53 layers are obtained using the aforementioned method. Figure 6 shows the areas of knowledge points corresponding to different layers in the figure, as shown in the highlighted part of the figure.

The second embodiment of the present invention designs a kind of visualization and quantitative analysis system to the feature expression ability of neural network, its structure is shown in Figure 7, and this system comprises:

(2) a visualization module configured to, based on the extracted sample-level features and region-level features, reduce its dimension to obtain a low-dimensional representation, and visualize the low-dimensional representation in the low-dimensional space;

It should be noted that those skilled in the art should understand that the functions of each module shown in the implementation of the above-mentioned visualization and quantitative analysis system for the middle-level feature expression ability of the neural network can refer to the aforementioned description of the middle-level feature expression ability of the neural network. Visualization and related descriptions of quantitative analysis methods can be understood. The functions of the modules shown in the implementation of the above-mentioned visualization and quantitative analysis system for the middle-level feature expression ability of the neural network can be realized by a program (executable instruction) running on the processor, or can be realized by a specific logic circuit. accomplish. In the embodiment of the present invention, if the above-mentioned visualization and quantitative analysis system for the expression ability of neural network middle layer is implemented in the form of software function modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention can be embodied in the form of software products in essence or the part that contributes to the prior art. The computer software products are stored in a storage medium and include several instructions for Make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the method in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), magnetic disk or optical disk. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.

Correspondingly, the embodiments of the present invention also provide a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, various method embodiments of the present invention are implemented. Computer-readable storage media includes both volatile and non-permanent, removable and non-removable media by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for computers include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

In addition, the embodiment of the present invention also provides a visualization and quantitative analysis system for the expression ability of the layer features in the neural network, which includes a memory for storing computer-executable instructions, and a processor; the processor is used to execute the memory in the memory The steps in the above-mentioned method implementations are implemented when computer-executable instructions are used. Among them, the processor can be a central processing unit (Central Processing Unit, referred to as "CPU"), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as "DSP"), application specific integrated circuits (Application Specific Integrated Circuit, referred to as "ASIC") and so on. The aforementioned memory may be a read-only memory ("ROM" for short), a random access memory (random access memory, "RAM" for short), a flash memory (Flash), a hard disk or a solid-state hard disk, and the like. The steps of the methods disclosed in the various embodiments of the present invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

It should be noted that in the invention documents of this patent, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without more limitations, an element defined by the statement "comprising a" does not exclude the presence of additional same elements in the process, method, article or device comprising the element. In the invention documents of this patent, if it is mentioned that an action is performed according to a certain element, it means that the action is performed based on at least the element, which includes two situations: performing the action only based on the element, and performing the action based on the element and Other elements perform the behavior. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 types, and 2 or more, 2 or more times, or 2 or more types.

All documents mentioned in the present invention are considered to be included in the disclosure content of the present invention in their entirety so as to serve as a basis for amendments when necessary. In addition, it should be understood that the above descriptions are only preferred embodiments of the present specification, and are not intended to limit the protection scope of the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the protection scope of one or more embodiments of this specification.

Claims

A method for visualizing and quantitatively analyzing neural network middle layer feature expression capabilities, characterized in that it includes the following steps

(1) Select the feature interpretation object:

Select the model to be analyzed, wherein the model includes a middle-level expression, including: a neural network, a hierarchical graph model;

(2) Extract neural network features:

Provide a set of input samples, input these samples into the above neural network, and extract the features of these samples, wherein the features include: sample-level features and region-level features;

(3) Feature dimensionality reduction to obtain visualization results:

First, the dimensionality reduction of the sample-level features is performed to obtain the visualization results of the sample-level features in the low-dimensional space; secondly, based on the low-dimensional representation of the sample-level features and the regional-level features, the dimensionality reduction of the regional-level features is performed to obtain the regional-level features Visualization results in low-dimensional space;

(4) Quantitative analysis of features based on visualization results:

Based on the visualization results, the quantity and quality of knowledge points in the features are quantitatively analyzed.
The method according to claim 1, wherein the extraction of sample level features in the step (2) further comprises the following steps:

Input a given set of samples into the neural network to be analyzed, and for each sample, extract the output features of an intermediate layer of the neural network to obtain the sample-level features corresponding to each input sample, and then the set of input The sample-level features that the sample corresponds to.
According to the method described in claim 1, it is characterized in that, in described step (2), to the extraction of regional level feature, further comprise the following steps:

Input a given set of input samples into the neural network to be analyzed, and for each sample, extract the output features of a certain convolutional layer of the neural network, so as to obtain the feature map corresponding to each input sample, where the feature map The high-dimensional vector corresponding to each position is the region-level feature of the sample in this region; when the height and width of this feature map are H and W respectively, and there are K channels in total, then this feature map contains HW region-level features, wherein each region-level feature is a K-dimensional vector.
According to the method described in claim 1, it is characterized in that, in the described step (3), the dimensionality reduction process to sample level feature comprises the following steps:

For each sample x corresponding to the sample-level features
through a projection matrix
Map it into a low-dimensional space to obtain a low-dimensional representation of sample-level features
And, optimize M such that the low-dimensional representation should satisfy the low-dimensional representation g and each class.
According to the method described in claim 4, it is characterized in that, the calculation of the degree of proximity between the low-dimensional representation and each category includes:

(a) Use radial distribution to model the distribution of low-dimensional representation g of sample-level features in low-dimensional space;

(b) Calculate how close the low-dimensional representation is to each category.
The method according to claim 5, wherein said step (a) comprises:

Based on radial distribution, the probability density function of g in low-dimensional space can be written as follows:

Among them, y∈{1,2,…,C} represents different categories in the classification task; π y represents the prior probability of the yth class; l g =‖g‖ represents the L2 norm of g, which is called the strength of g ; o g = g/l g represents the orientation of g; μ y represents the average direction of the yth class; κ( ) is a monotonically increasing function, and p(l g |y) represents the l The prior probability of g , p vMF (o g |μ y ,κ(l g )) represents the vMF distribution with mean direction μ y and aggregation parameter κ(l g ).
The method according to claim 5, wherein said step (b) comprises:

Based on the above radial distribution, and assuming that the prior probability of l g is independent of the category y, then the closeness Q M (y|x) between the low-dimensional representation g and the yth category is expressed as follows:
According to the method described in claim 1, it is characterized in that, in the described step (3), the dimensionality reduction process to region-level features further comprises the following steps:

For each sample x HW region-level features
through a projection matrix
Map them into a low-dimensional space to obtain a low-dimensional representation of HW region-level features
And, optimize Λ so that the low-dimensional representation should satisfy that the similarity between samples inferred based on the low-dimensional representation h={h (1) ,h (2) ,…,h (HW) } is the same as that inferred based on the network output The similarity between samples obtained should be as consistent as possible. Furthermore, the low-dimensional representation of region-level features needs to be aligned with the low-dimensional feature representation of sample-level features.
The method as claimed in claim 1, characterized in that, the knowledge point in the step (4) is a collection of region-level features that make the following formula greater than a certain threshold:

Among them, h (r) is the low-dimensional representation of the r-th region-level feature corresponding to a certain sample x; therefore, the knowledge point represents the region-level feature that makes max c p(y=c|h (r) )>τ, That is, the region-level features contained in the set {h (r) :max c p(y=c|h (r) )>τ}, where τ is a positive constant, and in a preferred example, the value of τ is The value range is 0.3-0.8.
A visualization and quantitative analysis system for neural network middle layer feature expression ability, characterized in that the system includes the following modules:

(1) The input module is configured as a pre-trained classification neural network and contains input samples of all possible categories;

(2) a feature extraction module configured to extract sample-level features and region-level features of the input samples;

(3) a visualization module configured to, based on the extracted sample-level features and region-level features, reduce its dimension to obtain a low-dimensional representation, and visualize the low-dimensional representation in the low-dimensional space;

(4) The quantitative analysis module is configured to quantitatively analyze the quantity and quality of knowledge points in the feature based on the visualization result of the feature at the region level.