CN112101522B - Interactive machine learning method based on visualization - Google Patents
Interactive machine learning method based on visualization Download PDFInfo
- Publication number
- CN112101522B CN112101522B CN202010842552.2A CN202010842552A CN112101522B CN 112101522 B CN112101522 B CN 112101522B CN 202010842552 A CN202010842552 A CN 202010842552A CN 112101522 B CN112101522 B CN 112101522B
- Authority
- CN
- China
- Prior art keywords
- data
- view
- training
- visualization
- visual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- User Interface Of Digital Computer (AREA)
- Image Generation (AREA)
Abstract
The invention discloses an interactive machine learning method based on visualization, which comprises the steps of abstracting neural network data into structural data, implicit value data and model evaluation data; carrying out visual mapping and coding through visual channels such as colors, positions, transparencies and the like according to the attributes of the data; designing a colored circular visual code to represent a neural network structure; designing visual coding fusing thermodynamic diagrams and parallel coordinates, and representing the change trend of the hidden state value; designing a time sequence confusion matrix visual code to represent a training result; the global presentation is completed by combining multi-view linkage, hidden state value analysis views based on thermodynamic diagrams and parallel coordinates are used as main views, model training can be evaluated by the aid of time sequence confusion matrix views, guesses can be verified quickly through network structure views, and new training data are generated for the next round of analysis. The three views have relatively independent functions and are mutually complementary, and the interactive machine learning method based on visualization is completed together.
Description
Technical Field
The invention relates to the technical field of information visualization analysis and machine learning, in particular to an interactive machine learning method based on visualization.
Background
Machine learning is the science of how to use computers to simulate or implement human learning activities, and is one of the most intelligent features in artificial intelligence, the most advanced research fields. In the big data era, along with the continuous increase of data analysis requirements of various industries, knowledge is efficiently acquired through machine learning, and the method becomes a main driving force for the development of the current machine learning technology gradually. How to carry out deep multi-dimensional analysis on complex and diverse data based on a machine learning method and obtain hidden, effective and understandable knowledge from massive data becomes a main direction of machine learning research under the current big data environment.
In selecting an appropriate machine learning model, it is often necessary to trade off the relationship between model accuracy and interpretability. Black box models (such as neural networks) usually have high accuracy, but the internal working mechanism of such models is difficult to understand, the importance of each feature on the model prediction result cannot be estimated, and the interaction relationship between different features is difficult to understand. Although many studies have been made to use machine learning models to achieve good results in a variety of problems, training a high performance model becomes a time consuming and trial and error requiring process if it is not clearly understood how and why a model operates.
The visualization technology combines the calculation and analysis capability of the machine excellence and the judgment and reasoning capability of the human excellence, uses visual elements to express data information, uses a human-computer interaction method to enhance the understanding, cognition and judgment of the human, makes up the deficiency of the machine in the deep analysis and prediction task, and makes full use of the intelligence of the human to effectively understand, diagnose and optimize the model. Due to the intuition and high efficiency of visualization technology, the visualization technology is more and more widely applied to scientific research and practical application.
However, the current research method related to machine learning and visualization technology still has shortcomings. Firstly, the training of the machine learning model is still a time-consuming and constantly trial and error process, the information contained in the training process is not well utilized by the improvement of the model, and a more intuitive and effective mode is lacked to help researchers improve the training process of the model. Secondly, in the existing work of visualizing the neural network, most of the work is directed to understanding and diagnosing the neural network. The information contained in the hidden state value is less searched, a certain rule is mostly found, the work of optimizing the neural network training according to the similarity of the hidden state value is lacked, and the change trends of the single neuron and the whole neuron can not be well reflected by the visual design of the hidden state value.
Disclosure of Invention
The invention aims to solve the technical problem of providing an interactive machine learning method based on visualization, aiming at a machine learning important branch of a neural network, fusing neural network structure data, neural network hidden state value data and model evaluation data after each training iteration in the training process, and interactively understanding, diagnosing and optimizing a machine learning model based on visualization.
In order to solve the technical problems, the invention adopts the technical scheme that:
an interactive machine learning method based on visualization, comprising the steps of:
s1: data abstraction
Collecting training data of a neural network, and abstracting the training data into three parts: neural network structure data, hidden state value data and model evaluation data;
s2: visualization mapping
Carrying out visual mapping on the data abstracted in the step S1 through a visual channel, wherein the neural network structure data adopts the shape of a circular mapping neuron, and the functional category of the neuron is coded by color; visual mapping of the hidden state value data combines thermodynamic diagrams and parallel coordinates; performing visual mapping on the model evaluation data after each training iteration by using a time sequence confusion matrix;
s3: visual layout
Carrying out specific visual layout and drawing realization on the mapping rule defined in the step S2; for the network structure view, iteratively calculating the neuron position of each layer of the network according to the network structure data and a layout formula; for the hidden state value analysis view, according to hidden state value data generated by a training sample, a coordinate position is calculated according to a layout formula, then parallel coordinate connecting lines and thermodynamic diagram rectangles are drawn, and the zoom degree and the transparency are introduced to fuse double views; for the time sequence confusion matrix view, according to the hidden state value data and the training result generated by each training, the coordinate position is calculated by a layout formula, the y-axis numerical values belonging to the same neuron are connected, and the corresponding interval regions are filled with the colors corresponding to the categories;
s4: interactive design
Rendering the above visualization module into an interactive visualization view by a rendering technology; the hidden state value analysis view is a main view, the training of the model can be evaluated through the assistance of the time sequence confusion matrix view, the guess can be quickly verified through the network structure view, and new training data are generated for the next round of analysis.
Further, in step S1, the data abstraction is specifically:
the abstract neural network structure data comprises the total number of layers of the neural network, the functional categories of the number of layers of each network and the number of neurons of each layer of network; the abstract neural network hidden state value data comprises a network layer sequence number, a neuron sequence number, a training time sequence number and hidden state value data generated by each neuron of a certain hidden layer on each training sample; the abstract model evaluation data comprises training time sequence numbers, classifier category sequence numbers, category total numbers and a prediction result statistical confusion matrix.
Further, in step S3, the visualization layout is implemented specifically as:
s3a: network structure view visualization layout implementation
Calculating a visual layout mode by combining a neural network structure data visual mapping method according to the neural network abstract data to realize the visual layout of the network structure view;
s3b: latent value analysis view visualization layout implementation
Calculating a visual layout mode by combining a neural network hidden value data visual mapping method according to the abstract data of the neural network hidden value, and realizing the visual layout of the hidden value analysis view;
s3c: time sequence confusion matrix view visualization layout implementation
And (4) according to the model evaluation abstract data after each training iteration, calculating a visual layout mode by combining a model evaluation data visual mapping method, and realizing the visual layout of the time sequence confusion matrix view.
Further, in step S3a, the network structure view visualization layout is implemented specifically as:
s3a1: defining structural data S of a neural network algorithm; determining the network layer serial number and the total network layer number of a neural network algorithm, and analyzing neurons contained in an input layer, a hidden layer and an output layer respectively;
s3a2: calculating the circular color of the neuron according to the position of the neuron;
s3a3: calculating the position coordinates of the starting point of each layer of neurons according to the number of the network layers and the number of the neurons in each layer;
s3a4: and sequentially drawing the neurons of each layer network and outputting a neural network structure view.
Further, in step S3b, the hidden-state-value-analysis-view visualization layout is specifically implemented as:
s3b1: calculating the width and height of a single rectangle according to the data set scale and the view size;
s3b2: reading the hidden state value data generated by each training sample, and calculating the x-axis value of the rectangular drawing starting point corresponding to the training sample;
s3b3: sequentially calculating the corresponding y-axis positions according to the sequence numbers of the neurons;
s3b4: calculating the color corresponding to the rectangle according to the hidden value number, and drawing a rectangular thermodynamic diagram;
s3b5: calculating the position of a broken line of a visual layer of the parallel coordinate according to the hidden state value data, wherein the x axis is shared with the thermodynamic diagram and represents the number of times of each training iteration;
s3b6: and sequentially calculating the corresponding y-axis position of each neuron according to the hidden state value of each neuron, and connecting the values belonging to the same neuron by using straight lines to draw a parallel coordinate broken line.
Further, in step S3c, the time-series confusion matrix view visualization layout is implemented specifically as:
s3c1: reading in a data set, and calculating an x-axis position corresponding to each training according to the hidden-state value data generated by each training;
s3c2: counting the numerical values of a true positive sample (TP, a correctly classified positive example), a false positive sample (FP, a wrongly classified positive example), a true negative sample (TN, a correctly classified negative example) and a false negative sample (FN, a wrongly classified negative example) according to the training result of each time, and calculating the y-axis position of each training;
s3c3: the y-axis values belonging to the same neuron are connected and the corresponding interval regions are filled with colors corresponding to the categories.
Compared with the prior art, the invention has the beneficial effects that:
1) Most of the existing work of visualizing the neural network aims at understanding and diagnosing the neural network, the information contained in the hidden state value is less explored, and the work of guiding the optimization of the hyper-parameters according to the change of the hidden state value is lacked. The method combines machine learning and visual analysis, and displays data such as a hidden state value, a loss function value, the accuracy of the model on a training set and the like after the neural network model is trained through a proper view, so as to provide a way for understanding the model training process. The abundant interactive means of the invention are convenient for analysis and exploration, find the reasons influencing the model effect or other valuable information in the model training, and provide the basis for the optimization of the training process.
2) The visual design of the change of the hidden state value only reflects one change trend of a single neuron or a whole neuron, and a visual method capable of highlighting two changes simultaneously is lacked. The design of the invention combines the thermodynamic diagram with the parallel coordinate, overcomes the problem of visual confusion caused by view fusion, and can simultaneously focus on the hidden state value change trend of the whole neuron or a single neuron on different training samples.
Drawings
Fig. 1 is a neural network structure visualization method.
FIG. 2 is a method for improved hidden state value visualization.
FIG. 3 is a method of time series confusion matrix visualization.
Fig. 4 is a schematic diagram of a neural network structure visualization method.
Fig. 5 is a schematic diagram of a coordinate system of a hidden value data visualization area.
FIG. 6 is a timing abstraction diagram for a confusion matrix.
Fig. 7 is a visualization-based interactive machine learning method framework.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The invention interactively understands, diagnoses and optimizes the machine learning model based on visualization by an effective information visualization method and combining a multi-view linkage strategy and a flexible interaction means. The technical scheme comprises the following steps: data abstraction, visual mapping and coding, visual layout implementation and interactive design are specifically as follows:
1. data abstraction
In order to quickly train a neural network model with a better classification effect, the method provided by the invention acquires training data of the neural network. The training data contains three parts: neural network structure data, hidden state value data and model evaluation data.
a) Neural network structure data:
the structural data defining the neural network algorithm is specifically defined as follows:
S＝{l 1 ,l 2 ,...,l i ,...,l m |1≤i≤m}
wherein i represents the network layer serial number of the neural network algorithm, m is the total network layer number of the algorithm, and l i Indicating a layer i network architecture. And, l 1 Input layer, l, representing neural network algorithms m Output layer network of the representation algorithm,/ 2 To l m-1 Is a hidden layer.
b) Neural network hidden value data:
defining the hidden state value data generated by a hidden layer containing m neurons in a neural network on n training samples as R, wherein the hidden state value data of the ith training sample on the hidden layer is t i ，v ij Representing the implicit value of the ith training sample on the jth neuron. Specific definitions of R are as follows:
R＝{t i ＝(v i1 ,v i2 ,...,v ij ,..,v im )|1≤i≤n,1≤j≤m}
c) Model evaluation data after each training iteration:
the classification result data generated by the n-times trained neural network algorithm is defined as V, and the detailed definition of V is shown as follows.
V＝{t i ＝(p i1 ,p i2 ,...,p ij ,...,p im )|1≤i≤n,1≤j≤m}
p ij ＝(c 1 ,c 2 ,...,c k ,...,c m )，1≤k≤m
Wherein i represents the serial number of training times, j represents the serial number of the category judged by the classifier, the total number of the categories is m, t i And (4) representing the prediction result generated by the classifier after the ith training. In addition, after combining the real class labels of the test set, training each prediction result p generated ij And may be represented by m real category data. For example,
this indicates the number of correctly classified positive examples, i.e., the number of classes that the classifier predicts as the jth class and the actual class is also the jth class.
2. Visualization mapping
a) Neural network structure data visualization mapping
As shown in fig. 1, the neural network structure visualization method uses red to represent the color of input neurons, gray to represent the color of hidden layer neurons, and green to represent the color of output neurons. The shape of the neuron is mapped by adopting a circle, and the general cognitive intuition of the neuron of human beings is met.
b) Implicit value data visualization mapping
The thermodynamic diagram and the parallel coordinates are combined in the visualization design, as shown in fig. 2, the thermodynamic diagram and the parallel coordinates can be mutually base diagrams of each other, the top views are switched through proper transparency setting, and the hidden state value change trend of the whole neuron or a single neuron on different training samples can be simultaneously noticed. Meanwhile, due to the combination of the two, an analyst can pay attention to the numerical attribute of the hidden state value at the same time, and can check the approximate change trend by means of the hidden state value of the color mapping.
In the top view based on parallel coordinates, each broken line represents the change trend of the hidden state value of one neuron. In position, the closer the point on each broken line is to the horizontal axis, the smaller the implicit state value represented by the point, and the larger the implicit state value represented by the point on each broken line is, and the relative position between the broken lines represents the actual adjacent relation of the neurons. In color, the broken line representing each neuron is gray, but the transparency is set, when the gray is gradually darker, the hidden state values of a large number of neurons are gathered at the gray, and the interaction of the matched views can be used for selecting the neuron corresponding to the part of the broken line.
In a bottom view based on the thermodynamic diagram, each long rectangle in the transverse and longitudinal directions shows a neuron state-hidden value change thermodynamic diagram. In terms of color, the magnitude of the hidden state value is mapped by using red, and the larger the hidden state value is, the more red the color of the hidden state value is, and otherwise, the more white the color of the hidden state value is. Positionally, the thermodynamic diagram longitudinal axis represents, in sequence from top to bottom, the actual order of the neurons, and this positional relationship allows the analyst to change through the interaction provided by the system, allowing him to put together neurons of "interest" for analysis.
The visual coding parts are explained by taking a top view as parallel coordinates and a bottom view as a thermodynamic diagram, and the coding mode after the two views are exchanged is not influenced.
c) Time series confusion matrix visualization mapping
A schematic diagram of the time-series confusion matrix visualization method is shown in fig. 3. In terms of color, each category to be classified is mapped by using one color, positive examples are represented by a green system, true positive samples are represented by dark green, and false positive samples are represented by light green; negative examples are represented by red color, true negative samples by dark red, and false negative samples by light red. In width, the number of samples is mapped using the width of the stream, with wider representing a greater number of samples. The total width of all streams at a certain number of training times is constant, being the sum of the number of all training samples.
3. Visual layout implementation
a) Visual layout implementation of neural network structure view
As shown in fig. 4, a single neuron is mapped by using a circle, the radius of the circle representing the neuron is defined as r, the lateral interval of the two neurons is padding _ n, and the interval between layers in the network structure is padding _ layer.
The visualization method has the following drawing idea for the neurons in each layer: the maximum number of neurons displayed transversely is defined as n _ max, and when the number of neurons at a certain layer is l i When the number of the neurons in the layer is larger than n _ max, the layer only draws n _ max-1 circles, and draws an ellipsis at the position of the n _ max/2 circle, which means that the number of the neurons in the layer is larger than n _ max, and the number of the neurons is smaller than n _ max, and then the layer is completely drawn. Under such a strategy, in order to ensure the beauty of the view (view is aligned in the center), the x-axis position of the drawing of the first circle is different due to the different numbers of neurons contained in the network layers, and the x-axis position of the drawing starting point of the neuron in each layer of the network is as follows:
the y-axis position of the origin of the mapping of the neurons in each layer of the network is as follows:
for the network layer l i For the j-th neuron in (1), its y-axis mapping position and its network layer
Similarly, the x-axis plot position is calculated as follows:
based on the above description, the specific implementation steps of the neural network structure visualization method are as follows:
the method comprises the following steps: reading a network structure data set S, and storing data in a variable arr;
step two: sequentially reading data of each layer from the variable arr, and calculating a drawing starting point of a neuron in each layer according to formulas (3-1) and (3-2);
step three: computing the network layer l according to equation (3-3) i X-axis size of all neurons in (a);
step four: for l i The network layer with the value greater than n _ max, wherein the n _ max/2 circle positions are drawn with an ellipsis to indicate that the number of neurons in the network layer is excessive;
step five: drawing l in sequence 1 To l m Neurons of a layer network;
step six: and outputting the neural network structure view.
b) Latent value analysis view visualization layout implementation
i. Time axis visual layout implementation based on parallel coordinates
The width of the area for displaying the hidden value data is defined as width _ h, the height is defined as height _ h, and the schematic diagram of the coordinate system of the display area is shown in fig. 5. For the ith training sample, t i The position of the horizontal axis (x-axis) on the view is calculated as follows:
where the purpose of i-1 is to make the drawing start point of the view the origin of coordinates.
Hidden state value v of jth neuron for ith training sample ij For example, the x-axis position is t i The position calculation mode of the vertical axis (y-axis) is shown in equations (3-6).
v max ＝maxv ij ,i∈[1,n],j∈[1,m] (3-5)
y ij ＝(v ij /v max )*height_h,i∈[1,n],j∈[1,m] (3-6)
The calculation method in the formula (3-6) is suitable for the case that the selected activation function of the neural network hidden layer is Sigmoid, and v in the hidden value data is ij Are all positive values, if v ij When both positive and negative values coexist, the calculation method of equation (3-6) is slightly changed.
In addition, the method for visualizing the time axis based on the parallel coordinates, which is implemented by the present invention, also supports the zoom operation, which is described above for the case that the zoom level zoom =1, which needs to show the whole data. When the zoom level is changed, the view is presented with data R by changing the focus data zoom The calculation method is as follows:
R zoom ＝R/zoom,zoom∈[1,8] (3-7)
there are 8 zoom levels, and a zoom operation changes the user's focused data set, but once the data set is determined, the way the corresponding variables are calculated does not change.
Based on the above description, the time axis visualization method based on parallel coordinates has the following specific implementation steps:
the method comprises the following steps: determining a focus data set R from a current view zoom level zoom zoom (calculated according to equation (3-7));
step two: reading a data set R zoom Storing the data in a variable arr;
step three: sequentially reading the hidden state value data t generated by each training sample from the variable arr i Calculating the x-axis position corresponding to the training according to a formula (3-4);
step four: to t according to the formula (3-6) i Each hidden state value v in ij Sequentially calculating the corresponding y-axis position y ij ；
Step five: connecting the implicit value data generated by the n training samples corresponding to the jth neuron by using a straight line;
step six: outputting the view at the current zoom level.
And if the user changes the zoom level, repeating the steps from the first step to the sixth step.
implementation of time axis visualization method based on thermodynamic diagram
The data used by the thermodynamic diagram-based time axis visualization method are consistent with those defined by the parallel coordinate-based time axis visualization method, the width of the data display area is also width _ h, and the height is height _ h. The method represents each hidden state value as a colored rectangle, and the drawing of each rectangle depends on five aspects of data: 1) an x-axis value of a rectangle drawing start point, 2) a y-axis value of a rectangle drawing start point, 3) a length of a rectangle, 4) a width of a rectangle, and 5) a color of a rectangle. The x-axis value of the rectangle drawing starting point is related to the sequence number of the training sample to which the rectangle drawing starting point belongs, and the calculation mode refers to the formula (3-4).
Hidden state value v of jth neuron for ith training sample ij For example, the y-axis position is calculated as follows:
y ij ＝(height_h/m)*j (3-8)
the rectangle is defined to be width _ rect and height _ rect, and the calculation mode is as follows:
defining a hidden value v ij Color _ rect ij The color "# FFF" (color _ min) is used to map the hidden state value 0, and "# F56C6C" (color _ max) is used to map the hidden state value 1. For colors with a hidden value between 0 and 1, the interpolation function for d3 is used for calculation, the formula of which is shown.
color_rect ij ＝d3.interpolate(color_min,color_max)(v ij ) (3-10)
Similar to the method of visualizing a time axis based on parallel coordinates, data used in the method based on thermodynamic diagrams is also changed according to a zoom operation of a user, and R is used when the zoom level is zoom zoom Representing the currently focused data set.
Based on the above description, the specific implementation steps of the thermodynamic diagram-based time axis visualization method are as follows:
the method comprises the following steps: determining a focus data set R from a current view zoom level zoom zoom (calculated according to equation (3-7));
step two: reading a data set R zoom Store data in the variablesIn amount arr;
step three: calculating the width _ rect and height _ rect of the rectangle according to the formula (3-9)
Step four: sequentially reading the implicit value data t generated by each training sample from the variable arr i Calculating an x-axis value of a rectangular drawing starting point corresponding to the training sample according to a formula (3-4);
step five: to t according to the formula (3-8) i Each hidden state value v in ij Sequentially calculating the corresponding y-axis position y ij ；
Step six: calculating v according to equation (3-10) ij Corresponding rectangle color, and drawing a rectangle;
step seven: and outputting the view at the current zoom level.
And if the user changes the zoom level, repeating the steps from the first step to the seventh step.
Fusion of parallel coordinates with thermodynamic diagram timeline visualization method
The method of the invention adopts a view overloading mode to fuse the parallel coordinates and the thermodynamic diagram time axis visualization, and the view is divided into: the first problem faced by this approach is visual confusion caused by view overlay, in which the top view and the bottom view are switched between a view based on parallel coordinates and a view based on thermodynamic diagrams according to the analysis requirements of an analyst. Aiming at the problem, a method for dynamically calculating the transparency of the parallel coordinate view according to the promiscuous degree of data is adopted. Data clutter is defined as follows:
definition of O init The current default transparency is parallel coordinate view, when the view is top view, the default transparency is 0.7, and the default transparency is 0.3 (the default value is obtained according to view debugging).
The implicit value data generated by the ith training sample is t i For arbitrary hidden state values v ij ∈[0,1]In this context, [0-1 ]]Dividing the interval into ten equal parts, counting the number of hidden state values contained in each equal part, and dividing the maximum value by m to obtain t i Data mixing degree of
R zoom The degree of mixing (c) is calculated as follows: />
d max The maximum value of the data clutter is represented, and the definition of the clutter is known as follows: d max ∈[0,1]。
Obtaining the maximum degree of mixing d of the data R max Then, the transparency of the parallel coordinate view is calculated as follows:
after the formula is used, the problem of visual confusion caused by dense numerical values can be improved to a great extent, and based on the above description, the improved hidden state value visualization method is realized by the following specific steps:
the method comprises the following steps: determining a focus data set R from a current view zoom level zoom zoom (calculated according to equation (3-7));
step two: calculating the transparency O of the parallel coordinate view according to the formula (3-12) final ；
Step three: drawing a top view according to the focus of attention, and then drawing a bottom view;
step four: and outputting the view at the current zoom level.
Repeating steps one through four when the user changes the zoom level or changes the view of interest.
c) Visual layout implementation of time sequence confusion matrix view
Based on the definition of the temporary classification result data generated in the neural network training process by the data abstraction part, V may be abstracted as shown in fig. 6 (a), in which the total classification m =3 is taken as an example for explanation. Viewed laterally in fig. 6 (a), each class is represented by a color system, the predicted true positive samples of the class after the current classifier training are represented by the dark color of the corresponding color system, and the false positives are represented by the light color of the color system.
As shown in fig. 6, the timing abstraction process of the confusion matrix comprises three main steps:
the method comprises the following steps: FIG. 6 (b) procedure, with classification result t of the first training 1 For example, the classification result is cut into rectangles in (b) according to the prediction category from the longitudinal direction, the width of each rectangle is consistent, the height is mapped to the corresponding numerical value, t 2 To t n So too does the segmentation of (2);
step two: FIG. 6 (c) Process, let t 1 To t n The rectangular bars predicted to be of class C1 are connected, as are classes C2 and C3;
step three: FIG. 6 (C) is a process of stacking "rivers" of the C1, C2 and C3 classes connected by the second step in the longitudinal direction;
then, the width of the area for displaying the hidden value data is defined as width _ v, the height is defined as height _ v, and the schematic diagram of the coordinate system of the display area is shown in fig. 1.
For the ith training, t i Position of the transverse (x-axis) axis on the view
The calculation is similar to equation (3-4). For each training, the y-axis position has m 2 The values need to be calculated, and the calculation formula is as follows.
Similar to the improved visual view of hidden state values, the analyst can also perform scaling operation on the time sequence confusion matrix view, and the focused data obtained after scaling is V zoom The calculation method is consistent with the formula (3-7).
Based on the above description, the time sequence confusion matrix visualization method is specifically implemented by the following steps:
the method comprises the following steps: determining a focus data set V from a current view zoom level zoom zoom (calculated according to equation (3-7));
step two: reading a data set V zoom Store the dataIn the variable arr;
step three: sequentially reading the hidden state value data t generated by each training from the variable arr i Calculating the x-axis position corresponding to the training according to a formula (3-4);
step four: sequentially calculating t according to the formula (3-13) i Each y-axis position in
Step five: connecting the y values belonging to the same neuron, and filling colors corresponding to the categories in the corresponding interval regions;
step six: and outputting the view at the current zoom level.
And if the user changes the zoom level, repeating the steps from the first step to the sixth step.
4. And (3) interaction and linkage design:
a) Neural network structure visualization method
As shown in fig. 1, the network structure visualization method is mainly divided into two parts: a control panel portion (fig. 1-a 1) and a view body portion (fig. 1-a 2). The control panel portion includes a panel that enables an analyst to express a guess and switches that control the retraining of the model. The view body part shows a visualization of the network structure of the initial algorithm, or the network structure readjusted by the analyst. The interactive design mainly comprises the following parts:
i. the network structure view adopts a mode of combining overview and detail, and allows an analyst to view the details of the network structure by using zooming operation;
when the analyst guesses the problems in the training process, the guess can be expressed in the form of a network structure through the control panel, and the view draws a new network structure view according to the input of the analyst;
using animation to show the hidden state value change process of the neuron, it is possible to choose whether a certain network or the whole network shows the animation together.
b) Improved hidden state value visualization method
As shown in fig. 2, the improved hidden state value visualization method mainly includes two parts: a control panel portion (fig. 2-b 1) and a view body portion (fig. 2-b 2). The control panel part (b 1) controls the display data resource of the view body, and can control the switching thermodynamic diagram and the parallel coordinates to be used as a top layer of the view. In addition, aiming at the condition that the number of training samples is too large and the available space of the screen is insufficient, the data are displayed in batches, and the data are switched by changing the range of the concerned data. The view body part (b 2) shows the change trend of the hidden state values of the whole neuron on different training samples, the top view of the view in fig. 2 is a thermodynamic diagram, and the bottom view is parallel coordinates. The interactive design mainly comprises the following parts:
i. when the top view is a parallel coordinate, when the mouse is suspended on a certain broken line, the broken line can be highlighted, the specific numerical value of the hidden value of the mouse sitting at the point can be displayed, and the long rectangle of the corresponding neuron in the bottom view can be highlighted;
when the top view is parallel coordinates, highlighting the plurality of broken lines selected by the frame, and highlighting the corresponding 'long rectangles' in the bottom view;
when the top view is a thermodynamic diagram, the mouse is highlighted as much as possible by the long rectangle in the direction of the transverse axis of the suspension, the hidden state value of the current mouse is displayed, and meanwhile, the corresponding fold line in the bottom view is highlighted;
when the top view is thermodynamic diagram, a single or a plurality of long rectangles in the direction of the transverse axis can be selected to move, and the longitudinal position of the long rectangles can be adjusted;
v. the view can show the variation trend of the hidden state values of all the neurons on different training samples through animation, and helps an analyst to know the variation process of the neurons through the animation.
c) Time sequence confusion matrix visualization method
The interactive design of the time sequence confusion matrix visualization method mainly comprises the following parts:
i. as shown in fig. 3, when the mouse is suspended on the flowsheet, the values of the true positive sample (TP), the false positive sample (FP), the true negative sample (TN), and the false negative sample (FN) of the training are displayed, and the circle with the same color system as the category is marked as the category boundary;
and ii, zooming the view through a mouse wheel, effectively helping an analyst focus on locally interested data when the data volume is too large, and dynamically changing the focused data by moving the 'window'.
Claims (6)
1. An interactive machine learning method based on visualization, comprising the steps of:
s1: data abstraction
Collecting training data of a neural network, and abstracting the training data into three parts: neural network structure data, hidden state value data and model evaluation data;
s2: visualization mapping
Carrying out visual mapping on the data abstracted in the step S1 through a visual channel, wherein the neural network structure data adopts the shape of a circular mapping neuron, and the functional category of the neuron is coded by color; visual mapping of the hidden state value data combines thermodynamic diagrams and parallel coordinates; performing visual mapping on the model evaluation data after each training iteration by using a time sequence confusion matrix;
s3: visual layout
Carrying out specific visual layout and drawing realization on the mapping rule defined in the step S2; for the network structure view, iteratively calculating the neuron position of each layer of the network according to the network structure data and a layout formula; for the hidden state value analysis view, according to hidden state value data generated by a training sample, a coordinate position is calculated according to a layout formula, then parallel coordinate connecting lines and thermodynamic diagram rectangles are drawn, and the zoom degree and the transparency are introduced to fuse double views; the visual integration of the parallel coordinates and the thermodynamic diagram time axis is realized in a view overloading mode, and the view is divided into: the device comprises a top view part and a bottom view part, wherein the view based on parallel coordinates and the view based on thermodynamic diagram are switched between the top view part and the bottom view part; for the time sequence confusion matrix view, according to the hidden state value data and the training result generated by each training, the coordinate position is calculated by a layout formula, the y-axis numerical values belonging to the same neuron are connected, and the corresponding interval regions are filled with the colors corresponding to the categories;
s4: interactive design
Rendering the network structure view, the hidden state value analysis view and the time sequence confusion matrix view into an interactive visual view by a rendering technology; the hidden state value analysis view is a main view, the training of the model can be evaluated through the assistance of the time sequence confusion matrix view, the guess can be quickly verified through the network structure view, and new training data are generated for the next round of analysis.
2. The visualization-based interactive machine learning method of claim 1, wherein in step S1, the data abstraction is specifically:
the abstract neural network structure data comprises the total number of layers of the neural network, the functional categories of the number of layers of each network and the number of neurons of each layer of network; the abstract neural network hidden state value data comprises a network layer sequence number, a neuron sequence number, a training time sequence number and hidden state value data generated by each neuron of a certain hidden layer on each training sample; the abstract model evaluation data comprises training time sequence numbers, classifier category sequence numbers, category total numbers and a prediction result statistical confusion matrix.
3. The visualization-based interactive machine learning method of claim 2, wherein in step S3, the visualization layout is implemented as:
s3a: network structure view visualization layout implementation
According to the abstract data of the neural network, a visual mapping method of the structural data of the neural network is combined to calculate a visual layout mode, and visual layout of a network structure view is achieved;
s3b: latent value analysis view visualization layout implementation
According to the abstract data of the hidden values of the neural network, a visual layout mode is calculated by combining a visual mapping method of the hidden values of the neural network, and the visual layout of the hidden value analysis view is realized;
s3c: time sequence confusion matrix view visualization layout implementation
And (4) according to the model evaluation abstract data after each training iteration, calculating a visual layout mode by combining a model evaluation data visual mapping method, and realizing the visual layout of the time sequence confusion matrix view.
4. The visualization-based interactive machine learning method of claim 3, wherein in step S3a, the network structure view visualization layout is implemented as:
s3a1: defining structural data S of a neural network algorithm; determining the network layer serial number and the total network layer number of a neural network algorithm, and analyzing neurons contained in an input layer, a hidden layer and an output layer respectively;
s3a2: calculating the circular color of the neuron according to the position of the neuron;
s3a3: calculating the position coordinates of the starting point of each layer of neurons according to the number of the network layers and the number of the neurons in each layer;
s3a4: and sequentially drawing the neurons of each layer network and outputting a neural network structure view.
5. The visualization-based interactive machine learning method of claim 4, wherein in step S3b, the hidden value analysis view visualization layout is implemented as:
s3b1: calculating the width and height of a single rectangle according to the data set scale and the view size;
s3b2: reading the hidden state value data generated by each training sample, and calculating the x-axis value of the rectangular drawing starting point corresponding to the training sample;
s3b3: sequentially calculating the corresponding y-axis positions according to the sequence numbers of the neurons;
s3b4: calculating the color corresponding to the rectangle according to the hidden value, and drawing a rectangular thermodynamic diagram;
s3b5: calculating the position of a fold line of a parallel coordinate view layer according to the hidden state data, wherein the x axis is shared with the thermodynamic diagram and represents the number of times of each training iteration;
s3b6: and sequentially calculating the corresponding y-axis position of each neuron according to the hidden state value of each neuron, and connecting the values belonging to the same neuron by using straight lines to draw a parallel coordinate broken line.
6. The visualization-based interactive machine learning method of claim 5, wherein in step S3c, the time-series confusion matrix view visualization layout is implemented as:
s3c1: reading in a data set, and calculating an x-axis position corresponding to each training according to the hidden-state value data generated by each training;
s3c2: counting the numerical values of the true positive sample, the false positive sample, the true negative sample and the false negative sample according to the result of each training, and calculating the y-axis position of each training;
s3c3: the y-axis values belonging to the same neuron are connected and the corresponding interval regions are filled with colors corresponding to the classes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010842552.2A CN112101522B (en) | 2020-08-20 | 2020-08-20 | Interactive machine learning method based on visualization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010842552.2A CN112101522B (en) | 2020-08-20 | 2020-08-20 | Interactive machine learning method based on visualization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112101522A CN112101522A (en) | 2020-12-18 |
| CN112101522B true CN112101522B (en) | 2023-04-18 |
Family
ID=73753957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010842552.2A Active CN112101522B (en) | 2020-08-20 | 2020-08-20 | Interactive machine learning method based on visualization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112101522B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114357253A (en) * | 2021-12-27 | 2022-04-15 | 上海商汤科技开发有限公司 | Data visualization method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103093643A (en) * | 2013-01-09 | 2013-05-08 | 东南大学 | Public parking lot berth quantity confirming method |
| CN107170023A (en) * | 2017-07-11 | 2017-09-15 | 四川大学 | A kind of method for visualizing developed towards the individual central site network of multivariate |
| CN107392085A (en) * | 2017-05-26 | 2017-11-24 | 上海精密计量测试研究所 | The method for visualizing convolutional neural networks |
| CN108280191A (en) * | 2018-01-25 | 2018-07-13 | 北京工商大学 | The comparison visual analysis method and system of more areas MRL standards |
| CN109643399A (en) * | 2016-08-09 | 2019-04-16 | 微软技术许可有限责任公司 | The interactive performance of multi-class classifier visualizes |
| WO2019092672A2 (en) * | 2017-11-13 | 2019-05-16 | Way2Vat Ltd. | Systems and methods for neuronal visual-linguistic data retrieval from an imaged document |
-
2020
- 2020-08-20 CN CN202010842552.2A patent/CN112101522B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103093643A (en) * | 2013-01-09 | 2013-05-08 | 东南大学 | Public parking lot berth quantity confirming method |
| CN109643399A (en) * | 2016-08-09 | 2019-04-16 | 微软技术许可有限责任公司 | The interactive performance of multi-class classifier visualizes |
| CN107392085A (en) * | 2017-05-26 | 2017-11-24 | 上海精密计量测试研究所 | The method for visualizing convolutional neural networks |
| CN107170023A (en) * | 2017-07-11 | 2017-09-15 | 四川大学 | A kind of method for visualizing developed towards the individual central site network of multivariate |
| WO2019092672A2 (en) * | 2017-11-13 | 2019-05-16 | Way2Vat Ltd. | Systems and methods for neuronal visual-linguistic data retrieval from an imaged document |
| CN108280191A (en) * | 2018-01-25 | 2018-07-13 | 北京工商大学 | The comparison visual analysis method and system of more areas MRL standards |
Non-Patent Citations (4)
| Title |
|---|
| Jaegul Choo et al..Visual Analytics for Explainable Deep Learning.《IEEE Computer Graphics and Applications》.2018,第38卷(第4期),84-92. * |
| Xiaoxiao Xiong et al..Visual potential expert prediction in question and answering communities.《Journal of Visual Languages & Computing》.2018,第48卷70-80. * |
| 朱敏等.面向城市空间热点分析的可视化方法综述.《计算机辅助设计与图形学学报》.2020,第32卷(第04期),551-567. * |
| 肖勇等.电网运行状态可视化综述.《计算机辅助设计与图形学学报》.2019,第31卷(第10期),1750-1758. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112101522A (en) | 2020-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Van Den Elzen et al. | Baobabview: Interactive construction and analysis of decision trees | |
| Kreuseler et al. | A flexible approach for visual data mining | |
| Yip et al. | Spatial aggregation: theory and applications | |
| Tino et al. | Hierarchical GTM: Constructing localized nonlinear projection manifolds in a principled way | |
| Symeonidis | Hand gesture recognition using neural networks | |
| CN112884021B (en) | Visual analysis system oriented to deep neural network interpretability | |
| US20220277192A1 (en) | Visual Analytics System to Assess, Understand, and Improve Deep Neural Networks | |
| Cao et al. | Real-time gesture recognition based on feature recalibration network with multi-scale information | |
| CN112101522B (en) | Interactive machine learning method based on visualization | |
| Maciejewski | Data representations, transformations, and statistics for visual reasoning | |
| Ma et al. | Visual analysis of class separations with locally linear segments | |
| Wu et al. | Feature-oriented design of visual analytics system for interpretable deep learning based intrusion detection | |
| Li et al. | GoTreeScape: Navigate and explore the tree visualization design space | |
| CN111598140A (en) | Remote sensing image classification method based on capsule network | |
| Eiter et al. | A Logic-based Approach to Contrastive Explainability for Neurosymbolic Visual Question Answering. | |
| König | Dimensionality reduction techniques for interactive visualization, exploratory data analysis, and classification | |
| Han et al. | A visualization model of interactive knowledge discovery systems and its implementations | |
| Pham et al. | Visualisation of fuzzy systems: requirements, techniques and framework | |
| Limonchik et al. | 3D model-based data augmentation for hand gesture recognition | |
| Vyas et al. | An interactive graphical visualization approach to CNNs and RNNs | |
| Einsfeld et al. | Knowledge generation through human-centered information visualization | |
| Gurevich et al. | Descriptive Image Analysis | |
| He | Dynamic prediction model and visual spatial analysis technology of big data | |
| Berthold et al. | Visualizing fuzzy points in parallel coordinates | |
| Krak et al. | The Technique of Inverse Multidimensional Scaling for the Synthesis of Machine Learning Models |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |