CN109660541B - Method and device for acquiring decentralized big data - Google Patents
Method and device for acquiring decentralized big data Download PDFInfo
- Publication number
- CN109660541B CN109660541B CN201811587682.5A CN201811587682A CN109660541B CN 109660541 B CN109660541 B CN 109660541B CN 201811587682 A CN201811587682 A CN 201811587682A CN 109660541 B CN109660541 B CN 109660541B
- Authority
- CN
- China
- Prior art keywords
- file
- data
- server
- symmetric encryption
- encryption key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000012545 processing Methods 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000009467 reduction Effects 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 5
- 238000004458 analytical method Methods 0.000 abstract description 4
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 238000004364 calculation method Methods 0.000 abstract description 2
- 238000013461 design Methods 0.000 abstract description 2
- 230000010354 integration Effects 0.000 abstract description 2
- 230000003993 interaction Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
- H04L63/045—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload wherein the sending and receiving network entities apply hybrid encryption, i.e. combination of symmetric and asymmetric encryption
Abstract
The invention provides a method and a device for acquiring decentralized big data, which do not need to distribute sensing data acquired by a gateway to different distributed servers according to tasks by a central server; the integration and analysis of the sensing data are not limited by the performance of the central server; meanwhile, in the process that the gateway and the distributed server carry out data interaction through the cloud storage server, the sensing data are encrypted through the combination of a symmetric encryption algorithm and an asymmetric encryption algorithm, and the authenticity of the sensing data is guaranteed; meanwhile, the gateway processes the sensing data by adopting the design idea of edge calculation, compresses the sensing data on the premise of ensuring the analysis value of the data, and improves the transmission and storage efficiency.
Description
Technical Field
The invention relates to the technical field of big data acquisition, in particular to a method and a device for acquiring decentralized big data.
Background
Big data, refers to a collection of data that cannot be captured, managed, and processed with conventional software tools in an affordable time frame.
Big data analysis is a technique of counting and analyzing all data for obtaining or predicting answers by means of known or unknown correlations between a plurality of data.
The big data analysis technology has higher application value in scenes such as smart cities, industrial systems and the like, but the scenes such as the smart cities are subdivided into a plurality of specific application scenes, such as smart traffic, smart agriculture, smart water affairs and the like, and sensing equipment suitable for or special application scenes is configured in each subdivided application scene. The sensing equipment of a plurality of application scenes generally collects sensing data through a gateway, integrates and analyzes the collected sensing data through a central processing platform, and then sends the integrated and analyzed sensing data to a distributed computer cluster for processing.
However, when there are many application scenarios and the sensing devices involved in each application scenario are complex, the processing efficiency of the central processing server on a large amount of sensing data is low, and the processing performance of the distributed computing cluster cannot be fully exerted.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method and a device for acquiring decentralized big data.
In a first aspect, an embodiment of the present invention provides a method for acquiring decentralized big data, where the method is applied to a gateway, and the method includes:
collecting data detected by a sensor;
processing the data to form processed data;
encrypting the processed data by using a symmetric encryption key to form an encrypted file;
selecting a preset asymmetric encryption public key to encrypt the symmetric encryption key, and producing a header file according to the serial number of the application scene of the processed data, wherein the asymmetric encryption public key and the header file form a combined file;
and transmitting the encrypted file and the combined file to a cloud storage server.
Further, processing the data to form processed data includes:
and constructing a data matrix from the data, and reducing the dimension of the data matrix into a dimension reduction dataset which is the processed data.
Further, the symmetric encryption key is randomly selected.
In a second aspect, an embodiment of the present invention further provides a method for acquiring decentralized big data, where the method is applied to a distributed server, and the method includes:
downloading the encrypted file and the combined file on the cloud server;
decrypting a symmetric encryption key by using the asymmetric encryption public key in the combined file;
and searching a server matched with the processed data according to the serial number in the head file in the combined file, and decrypting the encrypted file by using the symmetric encryption key.
Further, according to the serial number in the header file in the combined file, the server matched with the processed data is found, and the encrypted file is decrypted by using the symmetric encryption key, including:
and decrypting the encrypted file into a dimension-reduced data set by using the symmetric encryption key.
In a third aspect, an embodiment of the present invention further provides a device for acquiring decentralized big data, where the device is applied to a gateway, and the device includes:
the acquisition module is used for acquiring data detected by the sensor;
the processing module is used for processing the data to form processed data;
the encryption module is used for encrypting the processed data by using the symmetric encryption key to form an encrypted file;
the combination module is used for selecting a preset asymmetric encryption public key to encrypt the symmetric encryption key and producing a header file according to the serial number of the application scene of the processed data, and the asymmetric encryption public key and the header file form a combination file;
and the communication module is used for transmitting the encrypted file and the combined file to a cloud storage server.
Further, the processing module is further configured to construct a data matrix from the data, and the dimension reduction data matrix is a dimension reduction data set, and the dimension reduction data set is processed data.
In a fourth aspect, an embodiment of the present invention further provides a device for acquiring decentralized big data, where the device includes:
the downloading module is used for downloading the encrypted file and the combined file on the cloud server;
the decryption key module is used for decrypting a symmetric encryption key by using the asymmetric encryption public key in the combined file;
and the decryption module is used for finding the server matched with the processed data according to the serial number in the head file in the combined file and decrypting the encrypted file by using the symmetric encryption key.
Further, the decryption module is further configured to:
and decrypting the encrypted file into a dimension-reduced data set by using the symmetric encryption key.
In a fifth aspect, an embodiment of the present invention provides a computer storage medium for storing computer software instructions for the apparatus according to the third and fourth aspects.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a method and a device for acquiring decentralized big data, wherein a central server is not required to distribute sensing data acquired by a gateway to different distributed servers according to tasks; the integration and analysis of the sensing data are not limited by the performance of the central server; meanwhile, in the process that the gateway and the distributed server carry out data interaction through the cloud storage server, the sensing data are encrypted through the combination of a symmetric encryption algorithm and an asymmetric encryption algorithm, and the authenticity of the sensing data is guaranteed; meanwhile, the gateway processes the sensing data by adopting the design idea of edge calculation, compresses the sensing data on the premise of ensuring the analysis value of the data, and improves the transmission and storage efficiency.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flowchart of a method for collecting decentralized big data according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a method for collecting decentralized big data according to a second embodiment of the present invention;
fig. 3 is a block diagram of a decentralized big data acquisition device according to a third embodiment of the present invention;
fig. 4 is a block diagram of a decentralized big data acquisition device according to a fourth embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Referring to a flowchart of a method for collecting decentralized big data shown in fig. 1, the method is applied to a gateway, and specifically includes the following steps:
s101, collecting data detected by a sensor;
s102, processing the data to form processed data;
s103, encrypting the processed data by using a symmetric encryption key to form an encrypted file;
s104, a preset asymmetric encryption public key is selected to encrypt the symmetric encryption key, a header file is produced according to the serial number of the application scene of the processed data, and the asymmetric encryption public key and the header file form a combined file;
and S105, transmitting the encrypted file and the combined file to a cloud storage server.
Further, processing the data to form processed data includes:
and constructing a data matrix from the data, and reducing the dimension of the data matrix into a dimension reduction dataset which is the processed data.
Further, the symmetric encryption key is randomly selected.
Example two
Referring to fig. 2, a flow chart of a method for collecting decentralized big data is shown, where the method is applied to a distributed server, and specifically includes the following steps:
s201, downloading the encrypted file and the combined file on the cloud server;
s202, decrypting a symmetric encryption key by using the asymmetric encryption public key in the combined file;
s203, according to the serial number in the head file in the combined file, the server matched with the processed data is found, and the encrypted file is decrypted by using the symmetric encryption key.
Further, according to the serial number in the header file in the combined file, the server matched with the processed data is found, and the encrypted file is decrypted by using the symmetric encryption key, including:
and decrypting the encrypted file into a dimension-reduced data set by using the symmetric encryption key.
EXAMPLE III
For the method for acquiring decentralized big data provided in the foregoing embodiment, an embodiment of the present invention provides a device for acquiring decentralized big data, which is applied to a gateway, and referring to a structural block diagram of the device for acquiring decentralized big data shown in fig. 3, the device includes the following parts:
the acquisition module 31 is used for acquiring data detected by the sensor;
the processing module 32 is configured to process the data to form processed data;
the encryption module 33 is configured to encrypt the processed data by using the symmetric encryption key to form an encrypted file;
the combination module 34 is configured to select a preset asymmetric encryption public key to encrypt the symmetric encryption key, and produce a header file according to the number of the application scene of the processed data, where the asymmetric encryption public key and the header file form a combination file;
and the communication module 35 is configured to transmit the encrypted file and the combined file to a cloud storage server.
Further, the processing module 32 is further configured to construct a data matrix from the data, and reduce the dimension of the data matrix into a dimension reduction dataset, where the dimension reduction dataset is processed data.
Example four
For the method for acquiring decentralized big data provided in the foregoing embodiment, an embodiment of the present invention provides a device for acquiring decentralized big data, which is applied to a distributed server, and referring to a structural block diagram of the device for acquiring decentralized big data shown in fig. 4, the device includes the following parts:
a downloading module 41, configured to download the encrypted file and the combined file on the cloud server;
a decryption key module 42, configured to decrypt a symmetric encryption key using the asymmetric encryption public key in the combined file;
and a decryption module 43, configured to find a server matched with the processed data according to the number in the header file in the combined file, and decrypt the encrypted file using the symmetric encryption key.
Further, the decryption module 43 is further configured to:
and decrypting the encrypted file into a dimension-reduced data set by using the symmetric encryption key.
The embodiment of the invention also provides a computer storage medium for storing computer software instructions for the device provided by the embodiment.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
It should be noted that, in the embodiments provided in the present invention, it should be understood that the disclosed system and method can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments provided by the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for collecting decentralized big data is applied to a gateway, and the method comprises the following steps:
collecting data detected by a sensor;
processing the data to form processed data;
encrypting the processed data by using a symmetric encryption key to form an encrypted file;
selecting a preset asymmetric encryption public key to encrypt the symmetric encryption key, and producing a header file according to the serial number of the application scene of the processed data, wherein the asymmetric encryption public key and the header file form a combined file;
and transmitting the encrypted file and the combined file to a cloud storage server so that the distributed server downloads the encrypted file and the combined file from the cloud storage server, and finding a server matched with the processed data according to the serial number in the head file in the combined file.
2. The method of claim 1, wherein processing the data to form processed data comprises:
and constructing a data matrix from the data, and reducing the dimension of the data matrix into a dimension reduction dataset which is the processed data.
3. The method of claim 1, wherein the symmetric encryption key is randomly chosen.
4. A method for collecting decentralized big data is applied to a distributed server, and the method comprises the following steps:
downloading the encrypted file and the combined file on the cloud server;
decrypting a symmetric encryption key by using the asymmetric encryption public key in the combined file;
and searching a server matched with the processed data according to the serial number in the head file in the combined file, and decrypting the encrypted file by using the symmetric encryption key.
5. The method according to claim 4, wherein finding a server matching the processed data according to the number in the header file in the combined file, and decrypting the encrypted file using the symmetric encryption key comprises:
and decrypting the encrypted file into a dimension-reduced data set by using the symmetric encryption key.
6. An apparatus for collecting decentralized big data, applied to a gateway, the apparatus comprising:
the acquisition module is used for acquiring data detected by the sensor;
the processing module is used for processing the data to form processed data;
the encryption module is used for encrypting the processed data by using the symmetric encryption key to form an encrypted file;
the combination module is used for selecting a preset asymmetric encryption public key to encrypt the symmetric encryption key and producing a header file according to the serial number of the application scene of the processed data, and the asymmetric encryption public key and the header file form a combination file;
and the communication module is used for transmitting the encrypted file and the combined file to a cloud storage server so that the distributed server downloads the encrypted file and the combined file from the cloud storage server, and finds a server matched with the processed data according to the serial number in the head file in the combined file.
7. The apparatus of claim 6,
the processing module is further configured to construct a data matrix from the data, and the dimension reduction data matrix is a dimension reduction data set, and the dimension reduction data set is processed data.
8. A decentralized big data acquisition device, characterized in that it comprises:
the downloading module is used for downloading the encrypted file and the combined file on the cloud server;
the decryption key module is used for decrypting a symmetric encryption key by using the asymmetric encryption public key in the combined file;
and the decryption module is used for finding the server matched with the processed data according to the serial number in the head file in the combined file and decrypting the encrypted file by using the symmetric encryption key.
9. The apparatus of claim 8, wherein the decryption module is further configured to:
and decrypting the encrypted file into a dimension-reduced data set by using the symmetric encryption key.
10. A computer storage medium storing computer software instructions for use by the apparatus of any one of claims 6 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811587682.5A CN109660541B (en) | 2018-12-24 | 2018-12-24 | Method and device for acquiring decentralized big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811587682.5A CN109660541B (en) | 2018-12-24 | 2018-12-24 | Method and device for acquiring decentralized big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109660541A CN109660541A (en) | 2019-04-19 |
| CN109660541B true CN109660541B (en) | 2021-08-13 |
Family
ID=66115082
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811587682.5A Active CN109660541B (en) | 2018-12-24 | 2018-12-24 | Method and device for acquiring decentralized big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109660541B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111131379B (en) * | 2019-11-08 | 2021-06-01 | 西安电子科技大学 | Distributed flow acquisition system and edge calculation method |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102638480A (en) * | 2011-02-12 | 2012-08-15 | 苏州达联信息科技有限公司 | Method and device for processing monitoring data of railway track monitoring sensor network |
| CN102497556B (en) * | 2011-12-26 | 2017-12-08 | 深圳市云宙多媒体技术有限公司 | A kind of scene change detection method, apparatus, equipment based on time-variation-degree |
| KR101538424B1 (en) * | 2012-10-30 | 2015-07-22 | 주식회사 케이티 | Terminal for payment and local network monitoring |
| CN106405291A (en) * | 2016-09-27 | 2017-02-15 | 国网山东省电力公司梁山县供电公司 | Online monitoring and auxiliary equipment monitoring and analysis system based on power supply system |
| CN108066987B (en) * | 2016-11-18 | 2021-03-16 | 腾讯科技（深圳）有限公司 | Distribution method and device of application scene level combination and server |
| CN106878262B (en) * | 2016-12-19 | 2021-04-16 | 新华三技术有限公司 | Message detection method and device, and method and device for establishing local threat information library |
| CN107391929B (en) * | 2017-07-21 | 2020-07-28 | 北京粒创科技有限公司 | Virtual platform system based on user behavior data |
-
2018
- 2018-12-24 CN CN201811587682.5A patent/CN109660541B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109660541A (en) | 2019-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ghimire et al. | Recent advances on federated learning for cybersecurity and cybersecurity for federated learning for internet of things | |
| US10944795B2 (en) | Rating organization cybersecurity using active and passive external reconnaissance | |
| US11032323B2 (en) | Parametric analysis of integrated operational technology systems and information technology systems | |
| CN103782293B (en) | Multidimensional cluster for data partition | |
| EP3622402A1 (en) | Real time detection of cyber threats using behavioral analytics | |
| CN111477290A (en) | Federal learning and image classification method, system and terminal for protecting user privacy | |
| US20210281609A1 (en) | Rating organization cybersecurity using probe-based network reconnaissance techniques | |
| CN113542253B (en) | Network flow detection method, device, equipment and medium | |
| CN105610636B (en) | A kind of security log generation method towards cloud computing environment | |
| CN116108491B (en) | Data leakage early warning method, device and system based on semi-supervised federal learning | |
| CN113726783A (en) | Abnormal IP address identification method and device, electronic equipment and readable storage medium | |
| CN109660541B (en) | Method and device for acquiring decentralized big data | |
| CN115065555A (en) | Information security processing method and system | |
| CN110941823B (en) | Threat information acquisition method and device | |
| CN112437034A (en) | False terminal detection method and device, storage medium and electronic device | |
| CN112291277B (en) | Malicious software detection method, device, equipment and storage medium | |
| CN110909263B (en) | Method and device for determining companion relationship of identity characteristics | |
| US20220255956A1 (en) | Multi-stage network scanning | |
| CN116707927A (en) | Situation awareness method, system, computer equipment and storage medium | |
| JP6915457B2 (en) | Cyber attack information processing program, cyber attack information processing method and information processing equipment | |
| CN109617980B (en) | Method and device for analyzing decentralized big data | |
| CN112560085B (en) | Privacy protection method and device for business prediction model | |
| CN111565311B (en) | Network traffic characteristic generation method and device | |
| CN111506913B (en) | Audio encryption method and device, storage medium and electronic device | |
| CN117319089B (en) | Self-defense method and device of Internet of things based on semantic perception strategy self-generation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |