CN110443318B - Deep neural network method based on principal component analysis and cluster analysis - Google Patents
Deep neural network method based on principal component analysis and cluster analysis Download PDFInfo
- Publication number
- CN110443318B CN110443318B CN201910734831.4A CN201910734831A CN110443318B CN 110443318 B CN110443318 B CN 110443318B CN 201910734831 A CN201910734831 A CN 201910734831A CN 110443318 B CN110443318 B CN 110443318B
- Authority
- CN
- China
- Prior art keywords
- training
- neural network
- data
- cluster analysis
- pca
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 38
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000007621 cluster analysis Methods 0.000 title claims abstract description 24
- 238000000513 principal component analysis Methods 0.000 title claims abstract description 15
- 238000012549 training Methods 0.000 claims abstract description 45
- 238000012360 testing method Methods 0.000 claims abstract description 18
- 239000010410 layer Substances 0.000 claims abstract description 11
- 230000000694 effects Effects 0.000 claims abstract description 9
- 239000002356 single layer Substances 0.000 claims abstract description 7
- 238000004458 analytical method Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000008092 positive effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000012847 principal component analysis method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention relates to a deep neural network method based on principal component analysis and cluster analysis, which specifically comprises the following steps: s1: dividing the label data into training data and test data, wherein the training data is used for training and learning of a model; s2: performing feature dimension reduction on all initial data features of training data by using PCA, and extracting new main components; s3: K-Means cluster analysis is carried out on all training samples according to the principal components extracted by PCA; s4: and taking an upper layer data result as input, and combining the label obtained by clustering to form a single-layer neural network for training to obtain the network weight parameter. According to the deep neural network method based on principal component analysis and cluster analysis, the statistical feature learning method is combined with the neural network, the training mode of the traditional multi-layer neural network is optimized in the training process, and a better test effect is obtained for learning of a common deep neural network.
Description
Technical Field
The invention relates to the technical field of machine learning and artificial intelligence, in particular to a deep neural network method based on principal component analysis and cluster analysis.
Background
In recent years, artificial intelligence technology has attracted attention in both industry and academia, and machine learning methods play a central role in the field of artificial intelligence, and have been rapidly developed in many fields such as biometric sequences, natural language processing, computer vision, image recognition, financial market analysis, and the like. Among them, deep learning is remarkable in many fields. The deep belief network (Deep Belief Network, DBN) is used as a classical deep learning method, and has higher research significance in the aspects of feature extraction and classification learning.
The main idea of the Deep Belief Network (DBN) is divided into two stages of learning, wherein the first stage is to extract features by using an unsupervised pre-training (RBM) limited boltzmann machine (Restricted Boltzmann Machine), and the second stage is to combine the features obtained in the first stage and the labels corresponding to the data, and fine-tune model parameters by adopting a traditional Back propagation algorithm (BP). The first stage is similar to a feature abstraction process, for which the Cluster analysis and principal component analysis methods are both statistical-based feature extraction methods, and on the other hand, the Cluster analysis has a relatively low clustering efficiency for a large number of input features, so that in summary, the development of a method for embedding principal component analysis (Principle Component Analysis, PCA) and Cluster analysis (Cluster analysis) into a deep neural network model to perform unsupervised pre-training is urgent, and then the same BP algorithm is performed in the second stage to improve the training learning effect of the deep neural network.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides a deep neural network method based on principal component analysis and cluster analysis, which solves the problem of poor learning effect of the conventional common deep neural network.
The invention is realized by the following technical scheme:
a deep neural network method based on principal component analysis and cluster analysis specifically comprises the following steps:
s1: dividing an image set: dividing the label data into training data and test data, wherein the training data is used for training and learning of the model, and the test set is used for testing the comprehensive effect of the model;
s2: PCA feature dimension reduction: performing feature dimension reduction on all initial data features of training data by using PCA, and extracting new main components;
s3: K-Means cluster analysis: K-Means cluster analysis is carried out on all training samples according to the principal components extracted by PCA, the number of clusters is the number of hidden layers of the network structure, and the category of the clustering result is used as a hidden label corresponding to each training sample;
s4: taking the upper layer data result as input, and combining the label obtained by clustering to form a single-layer neural network for training to obtain a network weight parameter;
s5: the single-layer neural network weight obtained in the step S4 is used as the initial weight of the whole neural network, and then the weight is optimized by using a back propagation algorithm;
s6: the training model parameters described above are used for application prediction of the model.
Further, the PCA feature dimension reduction in S2 is to substantially reduce the initial feature dimension on the premise of retaining most of the information of the initial input.
Compared with the prior art, the invention has the beneficial effects that:
according to the deep neural network method based on principal component analysis and cluster analysis, the statistical characteristic learning method and the neural network are combined, the training mode of the traditional multi-layer neural network is optimized in the training process, a better test effect is obtained for learning of a common deep neural network, and meanwhile, the positive effect is achieved for training and learning of the deep neural network.
Drawings
Fig. 1 is a flowchart of an implementation of a deep neural network method based on principal component analysis and cluster analysis according to an embodiment of the present invention.
Detailed Description
The following examples are presented to specifically illustrate certain embodiments of the invention and should not be construed as limiting the scope of the invention. Modifications to the disclosure of the invention may be made in the materials, methods, and reaction conditions, all of which are intended to fall within the spirit and scope of the invention.
As shown in fig. 1, a deep neural network method based on principal component analysis and cluster analysis specifically includes the following steps:
s1: dividing an image set: dividing the label data into training data and test data, wherein the training data is used for training and learning of the model, and the test set is used for testing the comprehensive effect of the model;
s2: PCA feature dimension reduction: performing feature dimension reduction on all initial data features of training data by using PCA, and extracting new main components; for example, 784 dimension, wherein more than 95% of the principal component information is reserved through PCA analysis, and 60 dimension characteristics are obtained and used as variables of the next clustering analysis;
s3: K-Means cluster analysis: K-Means cluster analysis is carried out on all training samples according to principal components extracted by PCA, the number of clusters is the number of hidden layers of a network structure, the number of hidden layers is greater than or equal to 1, and the category of a clustering result is used as a hidden label corresponding to each training sample; for example, the network structure is 784-500-300-300-10, and the number of results of the first cluster is 500;
s4: taking the upper layer data result as input, and combining the label obtained by clustering to form a single-layer neural network for training to obtain a network weight parameter;
s5: the single-layer neural network weight obtained in the step S4 is used as the initial weight of the whole neural network, and then the weight is optimized by using a back propagation algorithm;
s6: the training model parameters described above are used for application prediction of the model.
The experimental procedure used four data sets (image classification) for training tests, the profile of which is given in table 1 below:
table 1 training data basic information
The experimental effect of the deep neural network method (PCA-KMeas-NN) based on principal component analysis and cluster analysis is better than that of a common deep neural network (DNN, which is trained only by BP algorithm), the error rate on the corresponding test data set is lower (the error rate is shown in the following table 2), and the PCA-KMeas-NN is better than DNN in terms of the error rate performance of four data sets in the test analysis.
TABLE 2 test results (test set error Rate)
| Data set | PCA-KMeans-NN | DNN |
| MNIST | 1.70％ | 1.98％ |
| smallMNIST | 3.50％ | 4.10％ |
| USPS | 2.46％ | 2.85％ |
| COIL20 | 0.68％ | 1.56％ |
Therefore, the method of the invention realizes the combined application of the statistical feature learning method and the neural network, optimizes the training mode of the traditional multi-layer neural network from the training process, and obtains better test effect aiming at the learning of the common deep neural network. The method provides a new learning method in the aspect of training the pre-training characteristics of the deep neural network, and has positive effects on training and learning of the deep neural network.
The foregoing description is only illustrative of the present invention and is not intended to limit the scope of the invention, and all equivalent structures or equivalent processes or direct or indirect application in other related technical fields are included in the scope of the present invention.
Claims (2)
1. The deep neural network method based on principal component analysis and cluster analysis is characterized by comprising the following steps of:
s1: dividing an image set: dividing the label data into training data and test data, wherein the training data is used for training and learning of the model, and the test set is used for testing the comprehensive effect of the model;
s2: PCA feature dimension reduction: performing feature dimension reduction on all initial data features of training data by using PCA, and extracting new main components;
s3: K-Means cluster analysis: K-Means cluster analysis is carried out on all training samples according to the principal components extracted by PCA, the number of clusters is the number of hidden layers of the network structure, and the category of the clustering result is used as a hidden label corresponding to each training sample;
s4: taking the upper layer data result as input, and combining the label obtained by clustering to form a single-layer neural network for training to obtain a network weight parameter;
s5: the single-layer neural network weight obtained in the step S4 is used as the initial weight of the whole neural network, and then the weight is optimized by using a back propagation algorithm;
s6: the training model parameters described above are used for application prediction of the model.
2. The method of deep neural network based on principal component analysis and cluster analysis according to claim 1, wherein the PCA feature dimension reduction in S2 is a substantial reduction of the initial feature dimension while retaining most of the initial input information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910734831.4A CN110443318B (en) | 2019-08-09 | 2019-08-09 | Deep neural network method based on principal component analysis and cluster analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910734831.4A CN110443318B (en) | 2019-08-09 | 2019-08-09 | Deep neural network method based on principal component analysis and cluster analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110443318A CN110443318A (en) | 2019-11-12 |
| CN110443318B true CN110443318B (en) | 2023-12-08 |
Family
ID=68434257
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910734831.4A Active CN110443318B (en) | 2019-08-09 | 2019-08-09 | Deep neural network method based on principal component analysis and cluster analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110443318B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112784499A (en) * | 2021-03-05 | 2021-05-11 | 云南电网有限责任公司电力科学研究院 | Electric reactor magnetic field simulation method based on deep learning instead of finite element analysis |
| CN113128031A (en) * | 2021-03-27 | 2021-07-16 | 南京林业大学 | Measuring device and evaluation method for comfort level of seat |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106600059A (en) * | 2016-12-13 | 2017-04-26 | 北京邮电大学 | Intelligent power grid short-term load predication method based on improved RBF neural network |
| CN109086805A (en) * | 2018-07-12 | 2018-12-25 | 华南理工大学 | A kind of clustering method constrained based on deep neural network and in pairs |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10713563B2 (en) * | 2017-11-27 | 2020-07-14 | Technische Universiteit Eindhoven | Object recognition using a convolutional neural network trained by principal component analysis and repeated spectral clustering |
-
2019
- 2019-08-09 CN CN201910734831.4A patent/CN110443318B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106600059A (en) * | 2016-12-13 | 2017-04-26 | 北京邮电大学 | Intelligent power grid short-term load predication method based on improved RBF neural network |
| CN109086805A (en) * | 2018-07-12 | 2018-12-25 | 华南理工大学 | A kind of clustering method constrained based on deep neural network and in pairs |
Non-Patent Citations (2)
| Title |
|---|
| 基于PCA与SOM神经网络算法的客户价值细分模型研究;胥学峰;邢德恩;宗炫君;;电气自动化(第03期);全文 * |
| 多信息融合技术在船舶柴油机故障诊断中的应用;尚前明;王瑞涵;陈辉;唐新飞;;中国航海(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110443318A (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111738301B (en) | Long-tail distribution image data identification method based on double-channel learning | |
| CN108846444B (en) | Multi-stage deep migration learning method for multi-source data mining | |
| CN107392241B (en) | Image target classification method based on weighted column sampling XGboost | |
| CN111581385B (en) | Unbalanced data sampling Chinese text category recognition system and method | |
| CN103854645B (en) | A kind of based on speaker's punishment independent of speaker's speech-emotion recognition method | |
| CN110443318B (en) | Deep neural network method based on principal component analysis and cluster analysis | |
| CN113378913B (en) | Semi-supervised node classification method based on self-supervised learning | |
| CN111506728B (en) | Hierarchical structure text automatic classification method based on HD-MSCNN | |
| CN114038037B (en) | Expression label correction and identification method based on separable residual error attention network | |
| CN112231477A (en) | Text classification method based on improved capsule network | |
| CN112232395B (en) | Semi-supervised image classification method for generating countermeasure network based on joint training | |
| CN110928981A (en) | Method, system and storage medium for establishing and perfecting iteration of text label system | |
| CN106548041A (en) | A kind of tumour key gene recognition methods based on prior information and parallel binary particle swarm optimization | |
| CN110516526A (en) | A kind of small sample target identification method based on Feature prototype metric learning | |
| CN110728144B (en) | Extraction type document automatic summarization method based on context semantic perception | |
| CN106951728B (en) | Tumor key gene identification method based on particle swarm optimization and scoring criterion | |
| CN112200797A (en) | Effective training method based on PCB noise labeling data | |
| CN114882531A (en) | Cross-domain pedestrian re-identification method based on deep learning | |
| CN111191033A (en) | Open set classification method based on classification utility | |
| CN107220320B (en) | Emerging technology identification method based on patent citation | |
| CN111984790B (en) | Entity relation extraction method | |
| CN113378563B (en) | Case feature extraction method and device based on genetic variation and semi-supervision | |
| CN104731890A (en) | Audio event classification method combining PLSA and AT | |
| CN110796260B (en) | Neural network model optimization method based on class expansion learning | |
| CN115223001A (en) | Medical image identification method based on transfer learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |