CN116578761B - Deep learning-based big data intelligent analysis method - Google Patents
Deep learning-based big data intelligent analysis method Download PDFInfo
- Publication number
- CN116578761B CN116578761B CN202310563553.7A CN202310563553A CN116578761B CN 116578761 B CN116578761 B CN 116578761B CN 202310563553 A CN202310563553 A CN 202310563553A CN 116578761 B CN116578761 B CN 116578761B
- Authority
- CN
- China
- Prior art keywords
- data
- model
- deep learning
- feature
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 25
- 238000013135 deep learning Methods 0.000 title claims abstract description 25
- 238000012549 training Methods 0.000 claims abstract description 34
- 238000007405 data analysis Methods 0.000 claims abstract description 20
- 239000013598 vector Substances 0.000 claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 15
- 230000000007 visual effect Effects 0.000 claims abstract description 13
- 238000013144 data compression Methods 0.000 claims abstract description 12
- 230000007246 mechanism Effects 0.000 claims abstract description 12
- 238000013136 deep learning model Methods 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 claims description 24
- 238000013527 convolutional neural network Methods 0.000 claims description 15
- 238000005516 engineering process Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 11
- 238000000605 extraction Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 7
- 238000013145 classification model Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 5
- 230000014509 gene expression Effects 0.000 claims description 4
- 238000003062 neural network model Methods 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000007794 visualization technique Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 17
- 238000007906 compression Methods 0.000 description 9
- 230000006835 compression Effects 0.000 description 8
- 230000006872 improvement Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 238000012800 visualization Methods 0.000 description 4
- 230000009467 reduction Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009960 carding Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013481 data capture Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000012954 risk control Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
- G06F18/15—Statistical pre-processing, e.g. techniques for normalisation or restoring missing data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to the technical field of data processing, in particular to a big data intelligent analysis method based on deep learning. Which comprises the following steps: raw data are acquired through a big data acquisition module, preprocessing is carried out, data features are acquired through a deep learning model, various types of feature vectors are selected through a feature selection algorithm and combined to obtain a prediction result, data training and classification are carried out through a stacked self-encoder model on a deep learning network architecture based on an attention mechanism, data analysis is carried out through a data compression algorithm, and the data are displayed in various visual modes. The invention fully covers each link, guarantees the objectivity of the result, also adopts excellent data processing tools such as a data compression algorithm and the like, provides guarantee for the high efficiency of data analysis, and has the advantages of clearer and more visual final result and further improved practicality.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a big data intelligent analysis method based on deep learning.
Background
With the popularization and development of mobile internet and internet of things, the generation and accumulation of large-scale data has become a global trend, but for such huge data volume, the traditional statistical method cannot meet the requirements of data analysis and exploration, and deep learning is widely focused on the powerful data analysis and processing capability;
however, in the current information technology field, the existing big data analysis method has some problems in practical application, for example, the traditional big data analysis algorithm cannot mine potential modes and rules in data, meanwhile, certain limitations exist on accuracy and precision of the algorithm for the problems of high dimensionality, low data quality and the like, and the prior art mostly focuses on data processing or focuses on optimization and training of a model, other important links in the data analysis process are ignored, the received data is disordered, and a large load is caused during transmission or processing, so that the stability of a system is influenced;
the intelligent analysis method for big data based on deep learning overcomes the defects of the traditional data mining and classifying algorithm, has higher accuracy and precision, can help users to better analyze and process data, and is applied to practical application in various fields.
Disclosure of Invention
The invention aims to provide a big data intelligent analysis method based on deep learning so as to solve the problems in the background technology.
In order to achieve the above purpose, the invention provides a big data intelligent analysis method based on deep learning, comprising the following steps:
s1, acquiring original data through a big data acquisition module, and accurately preprocessing the original data under the conditions of missing values, abnormal values and noise by adopting a big data preprocessing algorithm;
s2, obtaining data features by using a deep learning model, and selecting various types of feature vectors by using a feature selection algorithm to combine to obtain a prediction result;
s3, on a deep learning network architecture based on an attention mechanism, training and classifying data by using a stacked self-encoder model;
and S4, carrying out data analysis by adopting a data compression algorithm, and displaying analysis results by adopting various visual modes on the data.
As a further improvement of the technical scheme, when the original data is acquired, the big data acquisition module in the S1 automatically acquires various types of data by adopting a web crawler technology, and stores and sorts the data;
the web crawler technology comprises the following steps:
determining a target website and analyzing the website structure of the target website;
writing a web crawler program to process data grabbing and extracting of a target website;
acquiring target data according to the previously analyzed website layout and element information;
and storing the crawled data into a database.
As a further improvement of the present technical solution, the big data preprocessing algorithm in S1 includes the following steps: data is collected and checked, missing values are processed, abnormal values are processed, noise is processed, data sampling reduces the size of original data, data is reduced and transformed, data is standardized, a data set is split, and a model is verified.
As a further improvement of the technical scheme, the deep learning model in the step S2 uses a convolutional neural network model to perform data analysis in the feature extraction process, and extracts features;
the convolutional neural network model comprises the following steps:
extracting the original data preprocessed by the big data preprocessing algorithm;
defining a CNN model;
training a model;
and extracting the characteristics of the data by using the trained CNN model.
As a further improvement of the present technical solution, in the feature selection algorithm in S2, the feature selection process includes the following two steps:
selecting the most relevant features from each type of features to form a new feature vector;
the selected new feature vectors are classified by a learner.
As a further improvement of the present technical solution, the deep learning network architecture of the attention mechanism in S3 is implemented by the following algorithm:
let the input feature be { c×h×w }, where C represents the number of channels, H and W represent the height and width of the feature map, respectively, assuming that there are K attention headers, each header needs to output different weights, the weight of each header is defined as { k×c }, and input as x goes through K different convolution operations and corresponding weights to obtain K attention transformation results, namely:
{D*H*W}= Conv(x，wj), j=1,2,...,K
wherein D represents the output depth of each head, set to C/K, and splice the K outputs to obtain a weighted feature representation:
Concat(V1，V2，...，VK)={D1*H*W}
wherein d1=d×k, and the weighted features are sent to the subsequent layer for training, where the expression is:
alpha { i, j } = e (i, j)/sum of ownership scores
e（i，j）=f（hi，hj）
Where alpha { i, j } represents the attention weight of the ith row and jth column in the attention mechanism, e (i, j) is the element of the ith row and jth column in the attention score matrix, and f (hi, hj) represents the weighted sum of the feature vector of position i and the feature vector of position j in the input sequence.
As a further improvement of the present technical solution, the stacking self-encoder model in S3 includes the following steps:
dividing the data into a training set, a verification set and a test set according to the proportion according to the characteristics of the data weighting;
constructing a plurality of self-encoders, including an input layer, an encoding layer and a decoding layer, and performing feature learning in an unsupervised mode;
forming a deep neural network model from a plurality of self-encoders;
model training is carried out by using a training data set, and an optimization algorithm and a regularization method of self-adaptive learning rate are used for avoiding overfitting in the training process;
performing feature extraction on the data by using the trained stacked self-encoder model, and constructing a classification model;
and verifying the classification model by using the verification set test set, selecting a proper model and parameters, and predicting new data by using the trained model.
As a further improvement of the present technical solution, the data compression algorithm in S4 includes the following steps:
carrying out data coding on the predicted data and the analyzed data;
counting the occurrence frequency of symbols, sequencing the frequencies from small to large, combining the two smallest frequencies each time, constructing a tree, wherein the smaller frequency is represented by 0, the larger frequency is represented by 1, and outputting codes according to the sequence of leaf nodes;
and compressing and transmitting the encoded data.
As a further improvement of the present technical solution, the visualization manner in S4 includes: scatter plots, bar charts, histograms, bin charts, heat maps, and global relationship maps.
Compared with the prior art, the invention has the beneficial effects that:
according to the deep learning-based big data intelligent analysis method, all links of data processing, feature extraction, model training and result analysis are covered, a novel model structure and feature selection method are adopted, the objectivity of the result is guaranteed, excellent data processing tools such as a data compression algorithm are also adopted, the high efficiency of data analysis is guaranteed, the final result is clear and visual, and the practicability is further improved.
Drawings
FIG. 1 is a block diagram showing the overall structure of embodiment 1 of the present invention;
FIG. 2 is a block diagram of a web crawler technology flow according to embodiment 1 of the present invention;
FIG. 3 is a flow chart of a big data preprocessing algorithm in embodiment 1 of the present invention;
FIG. 4 is a flow chart of a convolutional neural network model of embodiment 1 of the present invention;
FIG. 5 is a block diagram of a feature selection algorithm according to embodiment 1 of the present invention;
fig. 6 is a flow chart of a data compression algorithm in embodiment 1 of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-6, embodiment 1 of the present invention provides a big data intelligent analysis method based on deep learning, comprising the following steps:
s1, acquiring original data through a big data acquisition module, accurately preprocessing the original data under the conditions of missing values, abnormal values and noise by adopting a big data preprocessing algorithm, and realizing operations such as sampling, reducing and transforming the data so as to ensure the high quality of the data and be suitable for deep learning training, reduce the occupation of the data in subsequent analysis, reduce the operation load and enable the data analysis to be more stable;
when the original data is acquired, the big data acquisition module in the S1 automatically acquires various types of data by adopting a web crawler technology, and stores and sorts the data;
as shown in fig. 2, the web crawler technology includes the following steps:
determining a target website, analyzing the website structure, knowing the page structure, elements, class names, labels, CSS selectors and the like of the website, wherein the information is helpful for creating programs to extract required data;
writing a web crawler program to process data capture and extraction of a target website, wherein Python is a common language, and a plurality of crawler frames and libraries are available for selection;
obtaining target data from previously analyzed web site layout and element information, which typically involves parsing an HTML page using regular expressions and obtaining the necessary data therefrom;
and storing the crawled data into a database.
As shown in fig. 3, the big data preprocessing algorithm in S1 includes the following steps:
collecting and checking data: firstly, raw data are required to be collected, and are subjected to carding and checking, so that the size, format, quality and the like of the data are known, wherein the situations include data loss, abnormal values, noise and the like;
missing value processing: for missing values, missing data records can be deleted and interpolation methods can be used for processing, such as methods of mean, median, model fitting and the like;
outlier processing: for outliers, two steps of detection and removal may be used. Detection uses methods such as statistical analysis, machine learning and the like, and cleaning can be performed in a modified or deleted mode;
noise treatment: for noise, filters may be used to smooth the data, such as mean filtering, median filtering, gaussian filtering, etc.;
data sampling reduces the original data size: when the data set is too large, a data sampling algorithm is used to reduce the original data size, in an attempt to keep the data distributed and reduce noise, such as random sampling and hierarchical sampling;
data reduction and transformation: for high-dimensional data, feature space and computational complexity can be reduced by using data dimension reduction, common data dimension reduction algorithms comprise PCA, LDA, t-SNE and the like, and in addition, original data can be transformed by using feature scaling, feature selection, feature transformation and the like, and model effect is improved.
Data normalization: normalization is a technique of scaling feature data to the same range, typically using statistical z-score transformations or minimum, maximum normalization, etc.;
splitting a data set: in real-scene applications, we need to divide the data set into a training set for building the model and a test set for verifying the model;
and (3) verifying a model: it is critical to verify the quality of the model, which can use various metrics such as classification accuracy, regression error, F1 score, ROC curve, etc. to determine the accuracy of the model.
S2, obtaining data features by using a deep learning model, obtaining high-quality interpretable features as input data, selecting various types of feature vectors by using a feature selection algorithm, combining the feature vectors to obtain a prediction result, and improving accuracy;
the deep learning model in the S2 uses a convolutional neural network model to conduct data analysis in the feature extraction process, and extracts features;
as shown in fig. 4, the convolutional neural network model includes the steps of:
extracting the original data preprocessed by the big data preprocessing algorithm;
defining a CNN model: the method comprises a convolution layer, an activation function, a pooling layer, a full connection layer and the like, and is designed according to the type of the characteristics and the characteristics of data;
model training: the method comprises model compiling, super parameter selecting, model training and the like, wherein the training aims at finding out the optimal weight and bias value;
the data is extracted by using a trained CNN model, and usually, the output of certain convolution layers can be selected as the needed characteristics, and then the required characteristics are sent into a full connection layer for classification or regression tasks.
As shown in fig. 5, in the feature selection algorithm in S2, let the sample set be d= { (x 1, y 1), (x 2, y 2), (xn, yn) }, where xi represents the feature vector of sample i, yi represents the corresponding category label, and xi contains m different types of features: xi= (x { i1}, x { i2}, x { im }, the process of feature selection comprises the following two steps:
selecting k most relevant features from each type of features to form a new feature vector, namely respectively selecting k most relevant features from m different types of features to generate a new feature vector tide { x = (tide { x }1, tide { x }2, tide { x } k), wherein tide { x }1, tide { x }2, tide { x } k respectively represent the selection results of m different types of features;
classifying the selected new feature vector tille { x } by a learner (such as a decision tree, a support vector machine, etc.);
in particular, the k features with highest correlation in each type of features can be selected by using a Wrapper method, and meanwhile, the importance degree between different features can be calculated and compared according to specific evaluation indexes (such as information gain, coefficient of kunity, average precision and the like) of the learner.
S3, on a deep learning network architecture based on an attention mechanism, data training and classification are carried out by using a stacked self-encoder model, and the architecture introduces the attention mechanism on the basis of a traditional deep learning network and can weight all the features so as to improve training precision and model popularization effect;
the deep learning network architecture of the attention mechanism in the step S3 is realized by the following algorithm:
let the input feature be { c×h×w }, where C represents the number of channels, H and W represent the height and width of the feature map, respectively, assuming that there are K attention headers, each header needs to output different weights, the weight of each header is defined as { k×c }, and input as x goes through K different convolution operations and corresponding weights to obtain K attention transformation results, namely:
{D*H*W}= Conv(x，wj), j=1,2,...,K
wherein D represents the output depth of each head, set to C/K, and splice the K outputs to obtain a weighted feature representation:
Concat(V1，V2，...，VK)={D1*H*W}
wherein d1=d×k, and the weighted features are sent to the subsequent layer for training, where the expression is:
alpha { i, j } = e (i, j)/sum of ownership scores
e（i，j）=f（hi，hj）
Wherein alpha { i, j } represents the attention weight of the ith row and jth column in the attention mechanism, e (i, j) is an element of the ith row and jth column in the attention score matrix, f (hi, hj) represents the weighted summation of the feature vector of the position i and the feature vector of the position j in the input sequence, the sum of all weight scores is used for normalizing the weight scores, the sum of all weights is ensured to be 1, the attention weight represents the contribution weight of different input positions to the target position, the weighted summation is used for calculating weighted average feature vectors, the most relevant information is extracted, and the attention mechanism network can adaptively adjust the weight of each position, so that the training precision and the model popularization effect are improved.
The stacking self-encoder model in S3 includes the steps of:
dividing the data into a training set, a verification set and a test set according to the proportion according to the characteristics of the data weighting, and carrying out pretreatment operations such as standardization, normalization and the like;
constructing a plurality of self-encoders, including an input layer, an encoding layer and a decoding layer, and performing feature learning in an unsupervised mode;
forming a deep neural network model by a plurality of self-encoders, namely stacking the self-encoders;
model training is carried out by using a training data set, and an optimization algorithm and a regularization method of self-adaptive learning rate are used for avoiding overfitting in the training process;
performing feature extraction on the data by using the trained stacked self-encoder model, and constructing a classification model, such as a machine learning model of SVM, KNN, logistic regression and the like or a neural network model;
the classification model is verified by using the verification set test set, a proper model and parameters are selected, new data are predicted by using the trained model, the controllability and stability of each link are ensured, and the problems of time and resource waste caused by repeated training for many times are avoided.
S4, data analysis is carried out by adopting a data compression algorithm, so that the bottleneck of large-scale data transmission and processing is avoided, the stability and processing speed of the system are improved, the high efficiency of data analysis is guaranteed, the analysis result is displayed by adopting various visual modes, the problems faced by large-scale data analysis can be comprehensively and effectively solved, the innovation is realized, the miniaturization and whole-course visualization are realized, and the high authorization rate and the practicability are realized;
as shown in fig. 6, the data compression algorithm in S4 includes the following steps:
performing data coding on the predicted data and the analyzed data, wherein the data coding comprises a series of codes based on information theory, such as error correction coding, markov chain coding, huffman coding, compressed index coding, entropy coding, lempel-Ziv-Welch coding and the like;
counting the occurrence frequency of symbols, sequencing the frequencies from small to large, combining the two smallest frequencies each time, and constructing a tree, wherein the smaller frequency is represented by 0, the larger frequency is represented by 1, and outputting codes according to the leaf node sequence, so that the volume of data transmission can be effectively reduced, and the transmission efficiency is improved;
the coded data is compressed and transmitted, common compression modes comprise lossless compression and lossy compression, wherein the lossless compression is to encode the data and remove redundant and repeated parts of the data according to redundancy and repeatability of the data, such as LZW, gzip and other lossless compression algorithms, the lossy compression is to allow partial data information to be lost in the compression process so as to replace higher compression ratio, and common lossy compression algorithms comprise JPEG, MPEG and the like, so that the high efficiency of data analysis is guaranteed through the data compression algorithm, the occupation of a system is reduced, and the operation stability is improved.
The visualization mode in S4 may better reveal the features and modes of the data, and extract useful information, including: a scatter plot showing the relationship between two numerical variables, wherein each point represents an observation; a bar graph for comparing different category variables, wherein each bar represents a category, and the height thereof represents a corresponding numerical value; the histogram is used for showing the distribution condition of the numerical variables, dividing the numerical range into a plurality of intervals, and the height of each interval represents the observed number falling into the interval; box line diagram: the method is used for displaying the distribution condition of numerical variables, five statistics of a data set are displayed in the form of boxes, the upper end and the lower end of each box respectively represent a first quartile and a third quartile, the median in each box is represented by a line, and points outside each box are abnormal values; heat map: for showing the relationship between the variables, the data is shown as different shades of color. Typically for analysis of multivariate data, there are typically two variables used to draw a graph, such as time and value; overall relationship diagram: for exhibiting an association of a plurality of variables, wherein each variable is represented as a different node and the relationship between variables is represented as an edge; map visualization: for presenting the distribution and relevance of geographic data, wherein each geographic location is represented as a point or area, and different colors, sizes, labels and symbols represent different features;
in conclusion, the visual presentation of the analysis result is carried out, so that the final result presentation is clearer and more visual, and the practicability is further improved.
In summary, the invention is represented according to the following examples (news data, mobile device user data):
(1) in S1, collecting news data on a network, processing the data by using a denoising and deduplication processing module, in S2, extracting text information of the news data by using a convolutional neural network model, in S3, performing data training and classification by using a stacked self-encoder model, classifying the news data according to a theme, in S4, performing visual analysis on a classification result by using an interactive visual technology, and enhancing a classification effect by adjusting model parameters;
(2) in S1, collecting mobile equipment user data, preprocessing such as data de-duplication and filtering, in S2, extracting mobile equipment user data characteristics such as user behaviors and position information by adopting a convolutional neural network model, in S3, training and classifying the data by using a stacked self-encoder model to realize accurate mobile end advertisement delivery, in S4, performing visual display on categories and analysis results by using an interactive visual technology, and continuously optimizing the model to improve advertisement delivery accuracy;
the invention has high innovativeness, provides a large data intelligent analysis method based on deep learning, has wide application range, can be widely applied to a plurality of fields such as mobile advertisement, financial risk control, medical image analysis and the like, provides a complete set of intelligent, all-dimensional and all-dimensional technical scheme, performs all-dimensional coverage on each link of data processing, feature extraction, model training and result analysis, adopts a novel model structure and feature selection method, improves the objectivity of the result, and also adopts excellent data processing tools such as a data compression algorithm and the like, thereby providing guarantee for the high efficiency of data analysis, and the final result is more clear and visual in presentation and further improves the practicability.
The foregoing has shown and described the basic principles, principal features and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the above-described embodiments, and that the above-described embodiments and descriptions are only preferred embodiments of the present invention, and are not intended to limit the invention, and that various changes and modifications may be made therein without departing from the spirit and scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (5)
1. The intelligent analysis method for big data based on deep learning is characterized by comprising the following steps:
s1, acquiring original data through a big data acquisition module, and accurately preprocessing the original data under the conditions of missing values, abnormal values and noise by adopting a big data preprocessing algorithm;
s2, obtaining data features by using a deep learning model, and selecting various types of feature vectors by using a feature selection algorithm to combine to obtain a prediction result;
s3, on a deep learning network architecture based on an attention mechanism, training and classifying data by using a stacked self-encoder model;
s4, carrying out data analysis by adopting a data compression algorithm, and displaying analysis results by adopting various visual modes on the data;
the deep learning model in the S2 uses a convolutional neural network model to conduct data analysis in the feature extraction process, and extracts features;
the convolutional neural network model performs data analysis and extracts features, and the method comprises the following steps:
extracting the original data preprocessed by the big data preprocessing algorithm;
defining a CNN model;
training a model;
extracting the characteristics of the data by using the trained CNN model;
in the feature selection algorithm in S2, the feature selection process includes the following two steps:
selecting the most relevant features from each type of features to form a new feature vector;
classifying the selected new feature vectors by a learner;
the deep learning network architecture of the attention mechanism in the step S3 is realized by the following algorithm:
let the input feature be { c×h×w }, where C represents the number of channels, H and W represent the height and width of the feature map, respectively, assuming that there are K attention headers, each header needs to output different weights, the weight of each header is defined as { k×c }, the input is x, and K attention transformation results are obtained through K different convolution operations and corresponding weights, that is:
{D*H*W}= Conv(x，wj), j=1,2,...,K
wherein D represents the output depth of each head, set to C/K, and splice the K outputs to obtain a weighted feature representation:
Concat(V1，V2，...，VK)={D1*H*W}
wherein d1=d×k, and the weighted features are sent to the subsequent layer for training, where the expression is:
alpha { i, j } = e (i, j)/sum of ownership scores
e（i，j）=f（hi，hj）
Where alpha { i, j } represents the attention weight of the ith row and jth column in the attention mechanism, e (i, j) is the element of the ith row and jth column in the attention score matrix, and f (hi, hj) represents the weighted sum of the feature vector of position i and the feature vector of position j in the input sequence;
the stacking self-encoder model in S3 includes the steps of:
dividing the data into a training set, a verification set and a test set according to the proportion according to the characteristics of the data weighting;
constructing a plurality of self-encoders, including an input layer, an encoding layer and a decoding layer, and performing feature learning in an unsupervised mode;
forming a deep neural network model from a plurality of self-encoders;
model training is carried out by using a training data set, and an optimization algorithm and a regularization method of self-adaptive learning rate are used for avoiding overfitting in the training process;
performing feature extraction on the data by using the trained stacked self-encoder model, and constructing a classification model;
and verifying the classification model by using the verification set test set, selecting a proper model and parameters, and predicting new data by using the trained model.
2. The deep learning-based big data intelligent analysis method according to claim 1, wherein: when the original data is acquired, the big data acquisition module in the S1 automatically acquires various types of data by adopting a web crawler technology, and stores and sorts the data;
the web crawler technology comprises the following steps:
determining a target website and analyzing the website structure of the target website;
writing a web crawler program to process data grabbing and extracting of a target website;
acquiring target data according to the previously analyzed website layout and element information;
and storing the crawled data into a database.
3. The deep learning-based big data intelligent analysis method according to claim 2, characterized by comprising the steps of: the big data preprocessing algorithm in the S1 comprises the following steps: data is collected and checked, missing values are processed, abnormal values are processed, noise is processed, data sampling reduces the size of original data, data is reduced and transformed, data is standardized, a data set is split, and a model is verified.
4. The deep learning-based big data intelligent analysis method according to claim 1, wherein: the data compression algorithm in S4 includes the following steps:
carrying out data coding on the predicted data and the analyzed data;
counting the occurrence frequency of symbols, sequencing the frequencies from small to large, combining the two smallest frequencies each time, constructing a tree, wherein the smaller frequency is represented by 0, the larger frequency is represented by 1, and outputting codes according to the leaf node sequence;
and compressing and transmitting the encoded data.
5. The deep learning-based big data intelligent analysis method according to claim 4, wherein: the visualization method in S4 includes: scatter plots, bar charts, histograms, bin charts, heat maps, and global relationship maps.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310563553.7A CN116578761B (en) | 2023-05-18 | 2023-05-18 | Deep learning-based big data intelligent analysis method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310563553.7A CN116578761B (en) | 2023-05-18 | 2023-05-18 | Deep learning-based big data intelligent analysis method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116578761A CN116578761A (en) | 2023-08-11 |
| CN116578761B true CN116578761B (en) | 2024-04-02 |
Family
ID=87533645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310563553.7A Active CN116578761B (en) | 2023-05-18 | 2023-05-18 | Deep learning-based big data intelligent analysis method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116578761B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109492830A (en) * | 2018-12-17 | 2019-03-19 | 杭州电子科技大学 | A kind of mobile pollution source concentration of emission prediction technique based on space-time deep learning |
| CN112434732A (en) * | 2020-11-17 | 2021-03-02 | 西安交通大学 | Deep learning classification method based on feature screening |
| CN114547598A (en) * | 2022-01-21 | 2022-05-27 | 江西理工大学 | Network intrusion detection method for improving BYOL (BYOL) self-supervision learning |
| CN115222994A (en) * | 2022-08-12 | 2022-10-21 | 河海大学 | Hyperspectral image classification method based on hybrid spectrum network and multi-head self-attention mechanism |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113627518B (en) * | 2021-08-07 | 2023-08-08 | 福州大学 | Method for realizing neural network brain electricity emotion recognition model by utilizing transfer learning |
-
2023
- 2023-05-18 CN CN202310563553.7A patent/CN116578761B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109492830A (en) * | 2018-12-17 | 2019-03-19 | 杭州电子科技大学 | A kind of mobile pollution source concentration of emission prediction technique based on space-time deep learning |
| CN112434732A (en) * | 2020-11-17 | 2021-03-02 | 西安交通大学 | Deep learning classification method based on feature screening |
| CN114547598A (en) * | 2022-01-21 | 2022-05-27 | 江西理工大学 | Network intrusion detection method for improving BYOL (BYOL) self-supervision learning |
| CN115222994A (en) * | 2022-08-12 | 2022-10-21 | 河海大学 | Hyperspectral image classification method based on hybrid spectrum network and multi-head self-attention mechanism |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116578761A (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209823B (en) | Multi-label text classification method and system | |
| CN110597735A (en) | Software defect prediction method for open-source software defect feature deep learning | |
| CN115471216B (en) | Data management method of intelligent laboratory management platform | |
| CN112287672A (en) | Text intention recognition method and device, electronic equipment and storage medium | |
| CN114612251A (en) | Risk assessment method, device, equipment and storage medium | |
| CN110188827A (en) | A kind of scene recognition method based on convolutional neural networks and recurrence autocoder model | |
| Kim et al. | Time series featurization via topological data analysis | |
| Yarragunta et al. | Prediction of air pollutants using supervised machine learning | |
| CN111898461A (en) | Time sequence behavior segment generation method | |
| KR20220090360A (en) | Device and method for analyzing and visualizing big data by controlling hyperparameters | |
| Tavakoli et al. | Clustering time series data through autoencoder-based deep learning models | |
| JP7440938B2 (en) | Event prediction system, event prediction method and program | |
| De Araujo et al. | Automatic cluster labeling based on phylogram analysis | |
| CN116578761B (en) | Deep learning-based big data intelligent analysis method | |
| CN111709437B (en) | Abnormal behavior detection method oriented to field process behavior of petrochemical industry | |
| CN117237095A (en) | Small micro financial industry risk early warning and monitoring platform based on big data analysis | |
| KR20210126378A (en) | Real-time sliding window based anomaly detection system for multivariate data generated by manufacturing equipment | |
| Feng et al. | Spatiotemporal prediction based on feature classification for multivariate floating-point time series lossy compression | |
| CN115809928A (en) | Project investment decision-making method and system | |
| CN113468203B (en) | Financial user image drawing method based on recurrent neural network and attention mechanism | |
| CN113705715A (en) | Time sequence classification method based on LSTM and multi-scale FCN | |
| CN114969511A (en) | Content recommendation method, device and medium based on fragments | |
| CN113643141A (en) | Method, device and equipment for generating explanatory conclusion report and storage medium | |
| CN113537240A (en) | Deformation region intelligent extraction method and system based on radar sequence image | |
| Karthik et al. | Detection of anomalies in time series data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240308 Address after: Room 30703, 7th Floor, Unit 3, Building 1, Hesheng Jingguang Center, No. 11 Tangyan Road, High tech Zone, Xi'an City, Shaanxi Province, 710075 Applicant after: Xi'an Yuntong Network Technology Co.,Ltd. Country or region after: China Address before: W57, 4th Floor, Lianhui Building, No. 5 Xingong Village, Sanlian Community, Longhua Street, Longhua District, Shenzhen City, Guangdong Province, 518110 Applicant before: Tulin Technology (Shenzhen) Co.,Ltd. Country or region before: China |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |