CN111582440A - Data processing method based on deep learning - Google Patents
Data processing method based on deep learning Download PDFInfo
- Publication number
- CN111582440A CN111582440A CN202010293921.7A CN202010293921A CN111582440A CN 111582440 A CN111582440 A CN 111582440A CN 202010293921 A CN202010293921 A CN 202010293921A CN 111582440 A CN111582440 A CN 111582440A
- Authority
- CN
- China
- Prior art keywords
- data
- deep learning
- error
- processing method
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000013135 deep learning Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 19
- 238000013528 artificial neural network Methods 0.000 claims abstract description 10
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 8
- 239000013598 vector Substances 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 230000007423 decrease Effects 0.000 claims description 3
- 238000009795 derivation Methods 0.000 claims description 2
- 230000004069 differentiation Effects 0.000 claims 1
- 230000009286 beneficial effect Effects 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000002087 whitening effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of data processing, and discloses a data processing method based on deep learning, which comprises the following steps: the method comprises the steps of firstly taking original data, preprocessing the original data, analyzing the structure of the data, processing the data to obtain finite element grid data, extracting data grid characteristics, converting three-dimensional data into a two-dimensional image, inputting the data into a neural network for forward propagation, obtaining data during deep learning of a user, obtaining key data used by the user, carrying out category distinguishing on the obtained key data, and obtaining a training sample based on the category distinguishing of the key data. According to the method, through preprocessing of the original data, classification distinguishing of the acquired data and establishment of the error model, the accuracy of the processed data can be effectively enhanced, the pertinence of the final processing result is enhanced by classification processing, and the data can be accurately and efficiently processed in real time.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data processing method based on deep learning.
Background
Data is a form of expression for facts, concepts, or instructions that may be processed by human or automated means. After the data is interpreted and given a certain meaning, it becomes information. The data processing is the collection, storage, retrieval, processing, transformation and transmission of data. The basic purpose of data processing is to extract and derive valuable, meaningful data for certain people from large, possibly chaotic, unintelligible amounts of data.
The existing data processing method mostly adopts a single neural network to process data, and the processing method not only has simple steps and low precision after data processing, but also causes insufficient and incomplete fine processing degree in the processing process.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides a data processing method based on deep learning, which solves the problems that the existing data processing method mostly adopts a single neural network to process data, and the processing method not only has simple steps and lower precision after data processing, but also has insufficient and incomplete refined processing degree in the processing process.
(II) technical scheme
In order to achieve the purpose, the invention provides the following technical scheme: a data processing method based on deep learning comprises the following steps:
s1: firstly, raw data is taken, the raw data is preprocessed, the structure of the data is analyzed, the data is processed to obtain finite element grid data, meanwhile, data grid characteristics are extracted, and three-dimensional data are converted into a two-dimensional image.
S2: the data are input into a neural network to be transmitted in the forward direction, data during deep learning of a user are obtained, key data used by the user are obtained, and the obtained key data are classified.
S3: and obtaining training samples based on the classification of the key data, and training the obtained class data by using a data target detection algorithm.
S4: establishing an error function model, inputting the obtained category data into the error function model for calculation, simultaneously setting a reference value, comparing with an expected value to obtain an error, judging the recognition degree through the error to obtain a user use opinion, and comparing with the reference value.
S4: the gradient vectors are determined by back propagation, and finally each weight is adjusted by the gradient vectors, adjusting towards the "score" a trend that makes the error tend to 0 or converge.
S5: the above process is repeated until the set number of times or the average value of the loss error no longer decreases.
S6: and obtaining the operation result of the data to be processed according to the finally obtained training results of different categories.
Preferably, the specific method for processing the data in the first step is to analyze and visualize the data.
Preferably, the method for detecting data in step three includes, but is not limited to, a method for training a YOLO-V3 network.
Preferably, the error function model is calculated in a regularization penalty mode to prevent overfitting.
Preferably, the smaller the error in the fourth step, the best opinion of the user is obtained, and the closest to the reference value is obtained.
Preferably, the back propagation is specifically a back derivative, an error function and each activation function in the neural network are required, with the ultimate goal of minimizing the error.
(III) advantageous effects
Compared with the prior art, the invention provides a data processing method based on deep learning, which has the following beneficial effects:
according to the data processing method based on deep learning, through preprocessing of original data, classification distinguishing of acquired data and establishment of an error model, accuracy after data processing can be effectively enhanced, pertinence of a final processing result is enhanced through classification processing, data can be accurately and efficiently processed in real time, and the problems that most of existing data processing methods adopt a single neural network to process data are solved.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example (b):
a data processing method based on deep learning comprises the following steps:
s1: the method comprises the steps of firstly taking original data, preprocessing the original data, analyzing the structure of the data, processing the data to obtain finite element grid data, extracting data grid characteristics, converting three-dimensional data into a two-dimensional image, preprocessing the data to include data normalization and data whitening, wherein the data normalization is one or more of simple scaling, sample-by-sample mean reduction and characteristic standardization.
S2: the data is input into a neural network for forward propagation, data during deep learning of a user is obtained, key data used by the user is obtained, the obtained key data are classified, and input values of each neuron are weighted and accumulated and then input into an activation function to serve as output values of the neuron.
S3: and obtaining training samples based on the classification of the key data, and training the obtained class data by using a data target detection algorithm.
S4: and establishing an error function model, inputting the obtained category data into the error function model for calculation, simultaneously setting a reference value, comparing the reference value with an expected value to obtain an error, judging the recognition degree through the error to obtain a user use opinion, and comparing the user use opinion with the reference value, so that the parameter amount is reduced, and the calculation amount is reduced to a certain extent.
S4: the gradient vectors are determined by back propagation, and finally each weight is adjusted by the gradient vectors, adjusting towards the "score" a trend that makes the error tend to 0 or converge.
S5: the above process is repeated until the set number of times or the average value of the loss error no longer decreases.
S6: and obtaining the operation result of the data to be processed according to the finally obtained training results of different categories.
Specifically, the specific method for processing the data in the step one is to analyze and visualize the data.
Specifically, the method for detecting data in step three includes, but is not limited to, a method for training a YOLO-V3 network, which is beneficial to the input, structure and output of the network.
Specifically, the error function model is calculated in a regularization punishment mode, and overfitting is prevented.
Specifically, the smaller the error in step four, the best the user opinion is obtained, and the closest to the reference value.
Specifically, back propagation is specifically the back derivation, the error function and each activation function in the neural network are required, with the ultimate goal of minimizing the error.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (6)
1. A data processing method based on deep learning is characterized by comprising the following steps:
s1: firstly, taking original data, preprocessing the original data, analyzing the structure of the data, processing the data to obtain finite element grid data, extracting the grid characteristics of the data, and converting three-dimensional data into a two-dimensional image;
s2: inputting data into a neural network for forward propagation, acquiring data during deep learning of a user, acquiring key data used by the user, and carrying out category differentiation on the acquired key data;
s3: obtaining training samples based on the classification of the key data, and training the obtained class data by using a data target detection algorithm;
s4: establishing an error function model, inputting the obtained category data into the error function model for calculation, simultaneously setting a reference value, comparing the reference value with an expected value to obtain an error, judging the recognition degree through the error to obtain a user use opinion, and comparing the user use opinion with the reference value;
s4: determining gradient vectors through back propagation, finally adjusting each weight value through the gradient vectors, and adjusting the error to be close to 0 or the convergence trend towards the score;
s5: repeating the above process until the average value of the set times or the loss error does not decrease;
s6: and obtaining the operation result of the data to be processed according to the finally obtained training results of different categories.
2. The data processing method based on deep learning of claim 1, wherein the specific method for processing the data in the first step is to analyze and visualize the data.
3. The deep learning-based data processing method of claim 1, wherein the data detection method in step three includes, but is not limited to, a YOLO-V3 network training method.
4. The data processing method based on deep learning of claim 1, wherein the error function model is calculated in a regularization penalty to prevent overfitting.
5. The deep learning-based data processing method as claimed in claim 1, wherein the smaller the error in the step four, the best user opinion is obtained, and the closest to the reference value is obtained.
6. The deep learning-based data processing method according to claim 1, wherein the back propagation is specifically required by a back derivation, an error function and each activation function in a neural network, and the final purpose is to minimize the error.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010293921.7A CN111582440A (en) | 2020-04-15 | 2020-04-15 | Data processing method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010293921.7A CN111582440A (en) | 2020-04-15 | 2020-04-15 | Data processing method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111582440A true CN111582440A (en) | 2020-08-25 |
Family
ID=72121142
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010293921.7A Pending CN111582440A (en) | 2020-04-15 | 2020-04-15 | Data processing method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111582440A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579582A (en) * | 2020-11-30 | 2021-03-30 | 贵州力创科技发展有限公司 | Data exploration method and system of data analysis engine |
| CN113177540A (en) * | 2021-04-14 | 2021-07-27 | 北京明略软件系统有限公司 | Positioning method and system based on trackside component |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105095962A (en) * | 2015-07-27 | 2015-11-25 | 中国汽车工程研究院股份有限公司 | Method for predicting dynamic mechanical property of material based on BP artificial neural network |
| WO2015180397A1 (en) * | 2014-05-31 | 2015-12-03 | 华为技术有限公司 | Method and device for recognizing data category based on deep neural network |
| CN109359608A (en) * | 2018-10-25 | 2019-02-19 | 电子科技大学 | A kind of face identification method based on deep learning model |
| CN109584290A (en) * | 2018-12-03 | 2019-04-05 | 北京航空航天大学 | A kind of three-dimensional image matching method based on convolutional neural networks |
| WO2019144521A1 (en) * | 2018-01-23 | 2019-08-01 | 杭州电子科技大学 | Deep learning-based malicious attack detection method in traffic cyber physical system |
| CN110766051A (en) * | 2019-09-20 | 2020-02-07 | 四川大学华西医院 | Lung nodule morphological classification method based on neural network |
-
2020
- 2020-04-15 CN CN202010293921.7A patent/CN111582440A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015180397A1 (en) * | 2014-05-31 | 2015-12-03 | 华为技术有限公司 | Method and device for recognizing data category based on deep neural network |
| CN105095962A (en) * | 2015-07-27 | 2015-11-25 | 中国汽车工程研究院股份有限公司 | Method for predicting dynamic mechanical property of material based on BP artificial neural network |
| WO2019144521A1 (en) * | 2018-01-23 | 2019-08-01 | 杭州电子科技大学 | Deep learning-based malicious attack detection method in traffic cyber physical system |
| CN109359608A (en) * | 2018-10-25 | 2019-02-19 | 电子科技大学 | A kind of face identification method based on deep learning model |
| CN109584290A (en) * | 2018-12-03 | 2019-04-05 | 北京航空航天大学 | A kind of three-dimensional image matching method based on convolutional neural networks |
| CN110766051A (en) * | 2019-09-20 | 2020-02-07 | 四川大学华西医院 | Lung nodule morphological classification method based on neural network |
Non-Patent Citations (2)
| Title |
|---|
| 杨军等: "基于深度体素卷积神经网络的三维模型识别分类", 《光学学报》 * |
| 沈萍等: "基于深度学习模型的花卉种类识别", 《科技通报》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579582A (en) * | 2020-11-30 | 2021-03-30 | 贵州力创科技发展有限公司 | Data exploration method and system of data analysis engine |
| CN113177540A (en) * | 2021-04-14 | 2021-07-27 | 北京明略软件系统有限公司 | Positioning method and system based on trackside component |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112507996B (en) | Face detection method of main sample attention mechanism | |
| CN110348486A (en) | Based on sampling and feature brief non-equilibrium data collection conversion method and system | |
| CN110084149A (en) | A kind of face verification method based on difficult sample four-tuple dynamic boundary loss function | |
| CN115994907B (en) | Intelligent processing system and method for comprehensive information of food detection mechanism | |
| CN111582440A (en) | Data processing method based on deep learning | |
| CN115953666B (en) | Substation site progress identification method based on improved Mask-RCNN | |
| CN111915101A (en) | Complex equipment fault prediction method and system based on LPP-HMM method | |
| CN110580510A (en) | clustering result evaluation method and system | |
| CN116340746A (en) | Feature selection method based on random forest improvement | |
| CN117315380B (en) | Deep learning-based pneumonia CT image classification method and system | |
| CN114037001A (en) | Mechanical pump small sample fault diagnosis method based on WGAN-GP-C and metric learning | |
| CN112365093A (en) | GRU deep learning-based multi-feature factor red tide prediction model | |
| CN112507881A (en) | sEMG signal classification method and system based on time convolution neural network | |
| CN110533636B (en) | Image analysis device | |
| Rahmat et al. | Tree identification to calculate the amount of palm trees using haar-cascade classifier algorithm | |
| CN113192629B (en) | Method and apparatus for automatic fetal heart interpretation | |
| CN115444419A (en) | Domain-adaptive intelligent emotion recognition method and device based on electroencephalogram signals | |
| CN114708634A (en) | Relative weight analysis method and device based on face image and electronic equipment | |
| CN114973307A (en) | Finger vein identification method and system for generating countermeasure and cosine ternary loss function | |
| CN113935413A (en) | Distribution network wave recording file waveform identification method based on convolutional neural network | |
| CN113283378A (en) | Pig face detection method based on trapezoidal region normalized pixel difference characteristics | |
| CN113011446A (en) | Intelligent target identification method based on multi-source heterogeneous data learning | |
| CN106599765A (en) | Method and system for judging living body based on continuously pronouncing video-audio of object | |
| CN111400685A (en) | Security identity authentication method adopting competition matching | |
| CN117689999B (en) | Method and system for realizing TC4 tape coiling process optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200825 |