CN109344333A - A kind of internet big data analysis extracting method and system - Google Patents
A kind of internet big data analysis extracting method and system Download PDFInfo
- Publication number
- CN109344333A CN109344333A CN201811315842.0A CN201811315842A CN109344333A CN 109344333 A CN109344333 A CN 109344333A CN 201811315842 A CN201811315842 A CN 201811315842A CN 109344333 A CN109344333 A CN 109344333A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- screening
- internet big
- keyword
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/606—Protecting data by securing the transmission between two devices or processes
Abstract
The invention discloses a kind of internet big data analysis extracting method and system, extracting method is analyzed the following steps are included: A, data acquisition unit acquire internet big data；B, the data of acquisition are sent to control unit after data screening module is screened and are handled；C, control unit sends to data characteristics extraction module and instructs, and carries out feature extraction according to pre-set keyword to the data after screening；D, the data after feature extraction store after being encrypted by DEU data encryption unit；E, the internet data after finally extracting is transmitted to background terminal by data transmission unit, the method that the present invention uses is easy to operate, and security performance is high, in addition, by being screened to data, feature extraction is carried out after screening, can be improved the analysis extraction efficiency and accuracy of internet big data, and reduces cost of labor, it improves work efficiency, furthermore it is possible to realize to take in row encryption after store, it is ensured that Information Security.
Description
Technical field
The present invention relates to data abstraction techniques field, specially a kind of internet big data analysis extracting method and system.
Background technique
People just know very well the importance of information since ancient times, and possessing more can obtain advantage with more accurately information, with
Society be constantly progressive, information is more and more diversified, and quantity is also growth at double, and the concept of big data also mentions therewith
Out, from numerous and jumbled data, required data is extracted and are analyzed, intuitive information is obtained, first chance is occupied with this, obtains benefit
Benefit.
Big data, or mass data, mass data;Be by enormous amount, structure is complicated, numerous types data are constituted
Data acquisition system is data processing and application model based on cloud computing, passes through the integrated shared of data, the intelligence of intersection multiplexing formation
Power resource and knowledge services ability.Research institution is so defined " big data ": " big data " is to need new tupe ability
With stronger decision edge, see clearly discovery power and process optimization ability magnanimity, high growth rate and diversified information assets.From
It says in a way, big data is the cutting edge technology of data analysis.In brief, it from the data of various type, quickly obtains
The ability for obtaining valuable information, is exactly big data technology.Internet big data analysis and extraction are generally by mutual in the prior art
Networking data platform is extracted and is analyzed automatically, although can be realized the extraction to internet big data and analysis,
Internet data platform feature is single, causes extraction efficiency low, and poor safety performance, be easy to cause leaking data, therefore, have
Necessity improves.
Summary of the invention
The purpose of the present invention is to provide a kind of internet big data analysis extracting method and systems, to solve above-mentioned background
The problem of being proposed in technology.
To achieve the above object, the invention provides the following technical scheme: a kind of internet big data analysis extracting method, divides
Analyse extracting method the following steps are included:
A, data acquisition unit acquires internet big data；
B, the data of acquisition are sent to control unit after data screening module is screened and are handled；
C, control unit to data characteristics extraction module send instruct, to the data after screening according to pre-set keyword into
Row feature extraction；
D, the data after feature extraction store after being encrypted by DEU data encryption unit；
E, the internet data after finally extracting is transmitted to background terminal by data transmission unit.
Preferably, data screening modular approach is as follows in the step B:
A, multiple noise datas are extracted from data to be screened as sample data；
B, conversion process is carried out to each sample data, obtains the transformation data of each sample data；
C, by preparatory trained data classification model, it is pre- that label is carried out to each sample data and each transformation data
It surveys, determines the target labels and target labels probability of each sample data；
D, according to the target labels of each sample data and target labels probability, each sample data is screened, obtains number of targets
According to.
Preferably, data ciphering method is as follows in the step D:
A, cleaning operation is carried out to data to be encrypted first；
B, the operation of AES encryption algorithm is carried out to the data after cleaning later, obtains an encrypted ciphertext data；
C, hyperchaos cryptographic calculation is carried out to a ciphertext data later again, obtains secondary ciphertext data；
D, finally secondary ciphertext data are carried out carrying out des encryption operation, the final encryption of complete paired data.
Preferably, the data characteristics extraction module feature extracting method is as follows:
A, data set is established, multiple Sub Data Sets to feature extraction are wherein included in data set；
B, feature training is carried out to data set, obtains training pattern；
C, the first keyword and the second keyword in data set are extracted；
D, each Sub Data Set in cyclic search data set, using the first keyword and the second keyword as primary condition, subdata
Collection scans for；
E, search is matched to the first keyword or the second keyword in each Sub Data Set, then extracts to data.
Preferably, a kind of internet big data analysis extraction system, including control unit, data acquisition unit, data sieve
Menu member, data characteristics extraction unit, data encryption storage unit and data outputting unit, the data acquisition unit pass through number
Control unit is connected according to screening unit, described control unit is separately connected data characteristics extraction unit, data encryption storage unit
And data outputting unit；Wherein, the data acquisition unit is for acquiring internet big data；The data screening unit is used for
Internet big data is screened, interference information is removed；The data characteristics extraction unit be used for the data after screening into
Row feature extraction；The data encryption storage unit is for carrying out encryption storage to the internet big data after feature extraction.
Compared with prior art, the beneficial effects of the present invention are:
(1) method that the present invention uses is easy to operate, and security performance is high, in addition, being carried out after screening by screening to data
Feature extraction, can be improved the analysis extraction efficiency and accuracy of internet big data, and reduce cost of labor, improve
Working efficiency, furthermore it is possible to realize to take in row encryption after store, it is ensured that Information Security.
(2) the data screening module that the present invention uses can carry out automatically data screening according to computer program, and it is convenient to operate
And it is time-consuming short, human resources can either be saved, and be able to ascend data screening efficiency, further improve the extraction effect of data
Rate.
(3) data ciphering method that the present invention uses can transmit data and carry out multi-enciphering, improve the peace of data
Full property and confidentiality.
It (4), can by the first keyword of search and the second keyword in the data characteristics extracting method that the present invention uses
It reduces and extracts difficulty, improve feature extraction precision.
Detailed description of the invention
Fig. 1 is flow chart of the present invention；
Fig. 2 is encryption method flow chart of the present invention；
Fig. 3 is apparatus of the present invention functional block diagram.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Fig. 1-3 is please referred to, the present invention provides a kind of technical solution: the invention provides the following technical scheme: a kind of internet
Big data analysis extracting method, analysis extracting method the following steps are included:
A, data acquisition unit acquires internet big data；
B, the data of acquisition are sent to control unit after data screening module is screened and are handled；
C, control unit to data characteristics extraction module send instruct, to the data after screening according to pre-set keyword into
Row feature extraction；
D, the data after feature extraction store after being encrypted by DEU data encryption unit；
E, the internet data after finally extracting is transmitted to background terminal by data transmission unit.
The present invention needs to use screening, feature extraction and the cryptographic operation to data, the present invention in analysis extraction process
Data screening module can carry out automatically data screening according to computer program, it is convenient and time-consuming short to operate, and can either save people
Power resource, and it is able to ascend data screening efficiency；Pass through the first keyword of search and the second keyword, energy in feature extracting method
Enough reduce extracts difficulty, improves feature extraction precision；Data ciphering method can transmit data and carry out multi-enciphering, improve
The safety and confidentiality of data by the processing of three above step can accelerate the analysis extraction efficiency of data, and
It can ensure the safety of data.
In the present invention, data screening modular approach is as follows in step B:
A, multiple noise datas are extracted from data to be screened as sample data；
B, conversion process is carried out to each sample data, obtains the transformation data of each sample data；
C, by preparatory trained data classification model, it is pre- that label is carried out to each sample data and each transformation data
It surveys, determines the target labels and target labels probability of each sample data；
D, according to the target labels of each sample data and target labels probability, each sample data is screened, obtains number of targets
According to.
The data screening module that the present invention uses can carry out automatically data screening according to computer program, operate convenient and consume
When it is short, human resources can either be saved, and be able to ascend data screening efficiency.
In the present invention, data ciphering method is as follows in step D:
A, cleaning operation is carried out to data to be encrypted first；
B, the operation of AES encryption algorithm is carried out to the data after cleaning later, obtains an encrypted ciphertext data；
C, hyperchaos cryptographic calculation is carried out to a ciphertext data later again, obtains secondary ciphertext data；
D, finally secondary ciphertext data are carried out carrying out des encryption operation, the final encryption of complete paired data.
The data ciphering method that the present invention uses can transmit data and carry out multi-enciphering, improve the safety of data
And confidentiality.
In the present invention, data characteristics extraction module feature extracting method is as follows:
A, data set is established, multiple Sub Data Sets to feature extraction are wherein included in data set；
B, feature training is carried out to data set, obtains training pattern；
C, the first keyword and the second keyword in data set are extracted；
D, each Sub Data Set in cyclic search data set, using the first keyword and the second keyword as primary condition, subdata
Collection scans for；
E, search is matched to the first keyword or the second keyword in each Sub Data Set, then extracts to data.
By the first keyword of search and the second keyword in the data characteristics extracting method that the present invention uses, can reduce
Difficulty is extracted, feature extraction precision is improved.
In addition, the invention also discloses a kind of internet big data analysis extraction system, including control unit 1, data are adopted
Collect unit 2, data screening unit 3, data characteristics extraction unit 4, data encryption storage unit 5 and data outputting unit 6, it is described
Data acquisition unit 2 connects control unit 1 by data screening unit 3, and described control unit 1 is separately connected data characteristics extraction
Unit 4, data encryption storage unit 5 and data outputting unit 6；Wherein, the data acquisition unit 2 is big for acquiring internet
Data；The data screening unit 3 removes interference information for screening to internet big data；The data characteristics mentions
Take unit 4 for carrying out feature extraction to the data after screening；After the data encryption storage unit 5 is used for feature extraction
Internet big data carries out encryption storage.
In conclusion the method that the present invention uses is easy to operate, security performance is high, in addition, by being screened to data,
Feature extraction is carried out after screening, can be improved the analysis extraction efficiency and accuracy of internet big data, and is reduced artificial
Cost improves work efficiency, furthermore it is possible to realize to take in row encryption after store, it is ensured that Information Security.
It although an embodiment of the present invention has been shown and described, for the ordinary skill in the art, can be with
A variety of variations, modification, replacement can be carried out to these embodiments without departing from the principles and spirit of the present invention by understanding
And modification, the scope of the present invention is defined by the appended.
Claims (5)
1. a kind of internet big data analysis extracting method, it is characterised in that: analysis extracting method the following steps are included:
A, data acquisition unit acquires internet big data；
B, the data of acquisition are sent to control unit after data screening module is screened and are handled；
C, control unit to data characteristics extraction module send instruct, to the data after screening according to pre-set keyword into
Row feature extraction；
D, the data after feature extraction store after being encrypted by DEU data encryption unit；
E, the internet data after finally extracting is transmitted to background terminal by data transmission unit.
2. a kind of internet big data analysis extracting method according to claim 1, it is characterised in that: in the step B
Data screening modular approach is as follows:
A, multiple noise datas are extracted from data to be screened as sample data；
B, conversion process is carried out to each sample data, obtains the transformation data of each sample data；
C, by preparatory trained data classification model, it is pre- that label is carried out to each sample data and each transformation data
It surveys, determines the target labels and target labels probability of each sample data；
D, according to the target labels of each sample data and target labels probability, each sample data is screened, obtains number of targets
According to.
3. a kind of internet big data analysis extracting method according to claim 1, it is characterised in that: in the step D
Data ciphering method is as follows:
A, cleaning operation is carried out to data to be encrypted first；
B, the operation of AES encryption algorithm is carried out to the data after cleaning later, obtains an encrypted ciphertext data；
C, hyperchaos cryptographic calculation is carried out to a ciphertext data later again, obtains secondary ciphertext data；
D, finally secondary ciphertext data are carried out carrying out des encryption operation, the final encryption of complete paired data.
4. a kind of internet big data analysis extracting method according to claim 1, it is characterised in that: the data characteristics
Extraction module feature extracting method is as follows:
A, data set is established, multiple Sub Data Sets to feature extraction are wherein included in data set；
B, feature training is carried out to data set, obtains training pattern；
C, the first keyword and the second keyword in data set are extracted；
D, each Sub Data Set in cyclic search data set, using the first keyword and the second keyword as primary condition, subdata
Collection scans for；
E, search is matched to the first keyword or the second keyword in each Sub Data Set, then extracts to data.
5. a kind of internet big data analysis extraction system, it is characterised in that: including control unit (1), data acquisition unit
(2), data screening unit (3), data characteristics extraction unit (4), data encryption storage unit (5) and data outputting unit (6),
The data acquisition unit (2) is separately connected by data screening unit (3) connection control unit (1), described control unit (1)
Data characteristics extraction unit (4), data encryption storage unit (5) and data outputting unit (6)；Wherein, the data acquisition is single
First (2) are for acquiring internet big data；The data screening unit (3) removes for screening to internet big data
Interference information；The data characteristics extraction unit (4) is used to carry out feature extraction to the data after screening；The data encryption is deposited
Storage unit (5) is for carrying out encryption storage to the internet big data after feature extraction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811315842.0A CN109344333A (en) | 2018-11-07 | 2018-11-07 | A kind of internet big data analysis extracting method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811315842.0A CN109344333A (en) | 2018-11-07 | 2018-11-07 | A kind of internet big data analysis extracting method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109344333A true CN109344333A (en) | 2019-02-15 |
Family
ID=65313821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811315842.0A Withdrawn CN109344333A (en) | 2018-11-07 | 2018-11-07 | A kind of internet big data analysis extracting method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344333A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110390204A (en) * | 2019-07-25 | 2019-10-29 | 上海应用技术大学 | Medical big data processing system and method |
| CN110765337A (en) * | 2019-11-15 | 2020-02-07 | 中科院计算技术研究所大数据研究院 | Service providing method based on internet big data |
| CN111556098A (en) * | 2020-04-08 | 2020-08-18 | 深圳供电局有限公司 | Artificial intelligence based analysis system and analysis method for internet of things data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202400A (en) * | 2016-07-11 | 2016-12-07 | 广东聚联电子商务股份有限公司 | A kind of big data collection and analysis method complaining behavior |
| CN106203171A (en) * | 2016-06-03 | 2016-12-07 | 中国电子科技网络信息安全有限公司 | Big data platform Security Index system and method |
| CN106897462A (en) * | 2017-03-13 | 2017-06-27 | 榆林学院 | Data statistic analysis plateform system |
| CN108040056A (en) * | 2017-12-15 | 2018-05-15 | 福州大学 | Safety medical treatment big data system based on Internet of Things |
-
2018
- 2018-11-07 CN CN201811315842.0A patent/CN109344333A/en not_active Withdrawn
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106203171A (en) * | 2016-06-03 | 2016-12-07 | 中国电子科技网络信息安全有限公司 | Big data platform Security Index system and method |
| CN106202400A (en) * | 2016-07-11 | 2016-12-07 | 广东聚联电子商务股份有限公司 | A kind of big data collection and analysis method complaining behavior |
| CN106897462A (en) * | 2017-03-13 | 2017-06-27 | 榆林学院 | Data statistic analysis plateform system |
| CN108040056A (en) * | 2017-12-15 | 2018-05-15 | 福州大学 | Safety medical treatment big data system based on Internet of Things |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110390204A (en) * | 2019-07-25 | 2019-10-29 | 上海应用技术大学 | Medical big data processing system and method |
| CN110765337A (en) * | 2019-11-15 | 2020-02-07 | 中科院计算技术研究所大数据研究院 | Service providing method based on internet big data |
| CN110765337B (en) * | 2019-11-15 | 2021-04-06 | 中科院计算技术研究所大数据研究院 | Service providing method based on internet big data |
| CN111556098A (en) * | 2020-04-08 | 2020-08-18 | 深圳供电局有限公司 | Artificial intelligence based analysis system and analysis method for internet of things data |
| CN111556098B (en) * | 2020-04-08 | 2023-09-15 | 深圳供电局有限公司 | Analysis system and analysis method for Internet of things data based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zou et al. | Encrypted traffic classification with a convolutional long short-term memory neural network | |
| CN104506484B (en) | A kind of proprietary protocol analysis and recognition methods | |
| CN102315974B (en) | Stratification characteristic analysis-based method and apparatus thereof for on-line identification for TCP, UDP flows | |
| CN105871832A (en) | Network application encrypted traffic recognition method and device based on protocol attributes | |
| CN109344333A (en) | A kind of internet big data analysis extracting method and system | |
| CN111082997B (en) | Network function arrangement method based on service identification in mobile edge computing platform | |
| CN111343169B (en) | System and method for gathering security resources and sharing information under industrial control environment | |
| CN109299742A (en) | Method, apparatus, equipment and the storage medium of automatic discovery unknown network stream | |
| CN101764704A (en) | Method for auditing internet sensitive contents and device thereof | |
| CN112019500B (en) | Encrypted traffic identification method based on deep learning and electronic device | |
| CN110460510A (en) | A kind of method, apparatus that establishing multi-conference, electronic equipment and medium | |
| CN105471635B (en) | A kind of processing method of system log, device and system | |
| CN109359686A (en) | A kind of user's portrait method and system based on Campus Network Traffic | |
| CN109284319A (en) | A kind of auditing system based on big data visualization technique | |
| CN109542867A (en) | Distribution type data collection method and device | |
| CN107529190B (en) | User data acquisition system and method | |
| CN114996207A (en) | Big data analysis method and system based on 5G cloud computing | |
| He et al. | Identification of SSH applications based on convolutional neural network | |
| CN110708341B (en) | User behavior detection method and system based on remote desktop encryption network traffic mode difference | |
| CN112800140A (en) | High-reliability data acquisition method based on block chain prediction machine | |
| CN109995784A (en) | A kind of data extraction accelerated method based on UDP | |
| CN106973314B (en) | Instruction identification method and system for network interaction | |
| CN112988829A (en) | Big data analysis processing system | |
| CN111126762A (en) | Intelligent workflow engine for electric power cloud security | |
| Zheng et al. | Identification of Malicious Encrypted Traffic Through Feature Fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190215 |