WO2019006640A1

WO2019006640A1 - Big data management system

Info

Publication number: WO2019006640A1
Application number: PCT/CN2017/091590
Authority: WO
Inventors: 陈钦鹏
Original assignee: 深圳齐心集团股份有限公司
Priority date: 2017-07-04
Filing date: 2017-07-04
Publication date: 2019-01-10

Abstract

A big data management system comprises a data collection unit, a data parallel processing unit, a cloud storage unit, a cloud storage encryption and decryption unit, a data access permission authentication unit, a data redundancy determining unit, and a data duplication deletion unit. The cloud storage unit comprises a local memory and a cloud memory. The cloud storage encryption and decryption unit is separately connected to the local memory, the cloud memory and the data access permission authentication unit. The data redundancy determining unit is separately connected to the data collection unit, the local memory and the cloud memory. The data duplication deletion unit is separately connected to the local memory, the cloud memory and the data redundancy determining unit. By means of the system, the processing efficiency and the using speed of big data are improved, a complete security protection system is also established, and the security management of the whole system is implemented.

Description

Big data management system

Technical field

The invention belongs to the field of big data technology, and in particular relates to a big data management system.

Background technique

With the rapid development of computer technology, the data of various fields and fields have grown rapidly. These data come from aspects, from sensors that collect weather conditions, digital photos, online video materials, to online shopping transaction records, to mobile phone GPS signals, and more. Along with the rapid expansion of data scale, the accumulated amount of data in various industries is getting larger and larger, the data types are more and more, and the data structure is more and more complex. It has surpassed the traditional data management system and the ability of processing modes. The traditional string Row database systems have been difficult to adapt to this rapidly growing application demand, showing significant lack of capacity in production practices, unable to meet the data management needs of the big data era.

Big data, or huge amount of data, means that the amount of data involved is so large that it cannot be absorbed, managed, processed, and organized in a reasonable time through the current mainstream software tools to help enterprises make more positive decisions. Information.

The rapid development of cloud computing technology puts forward higher requirements for big data management, especially for data centers in the production field. The amount of data has exceeded 100 million points, and the large proportion of storage is real-time and quasi-real-time production process data. .

technical problem

In order to overcome the technical problems existing in the prior art, the embodiment of the present invention provides a big data management system, which not only improves the processing efficiency and the use speed of big data, but also establishes a complete security protection system and implements the entire system. Security management.

Technical solution

The embodiment of the present invention is implemented as follows: a big data management system, comprising: a data collection unit, a data parallel processing unit, a cloud storage unit, a cloud storage encryption and decryption unit, a data access authority authentication unit, a data redundancy judgment unit, and data. Deleting the unit; the cloud storage unit includes a local storage and a cloud storage; the cloud storage encryption and decryption unit is respectively connected to the local storage, the cloud storage, and the data access authority authentication unit; and the data redundancy determination unit respectively The data collection unit, the local storage, and the cloud storage connection; the data deduplication unit is respectively connected to the local storage, the cloud storage, and the data redundancy judging unit.

Preferably, the system further comprises:

One or more clients, the client is connected to the cloud storage encryption and decryption unit, and by accessing the cloud storage unit, determining whether there is permission to access the target data point and obtaining the location information of the cloud storage unit to which the target data point belongs or directly Visit the location letter The cloud storage unit corresponding to the information; and the location information of the cloud storage unit to which the accessed target data point belongs.

Preferably, the data parallel processing unit adopts a parallel data preprocessing technology, and has a Map/Reduce processing model. Each of the processing tasks is processed in parallel by multiple Map tasks by calling a Map function, and the Map tasks are assigned to the processing. Execute on the execution node of the task assignment, and then call the Reduce function to merge the processing results of each Map task of each processing task to complete the data preprocessing.

Preferably, the cloud storage unit employs a discrete distributed data storage end.

Preferably, the discrete distributed data storage end comprises an HBase structure database and an HDFS structure database.

Preferably, the discrete distributed data storage end comprises: an MPP relational database and a Hadoop structure database.

Preferably, the system further comprises:

A data integration unit for integrating discrete source data and storing the consolidated source data to a cloud storage unit.

Preferably, the cloud storage encryption and decryption unit comprises a cloud storage security encryption chip and a cloud storage security decryption chip, and the security encryption chip and the security decryption chip adopt a secure socket layer SSL and/or a secure transmission layer TLS.

Preferably, the secure encryption chip and the secure decryption chip comprise one or more security processing units of AES, RSA, SHA, OTP, RNG, GUID.

Preferably, the data access authority authentication unit performs encryption and decryption authentication on the data by using an RSA method.

Beneficial effect

The big data management system provided by the embodiment of the invention not only improves the processing efficiency and the use speed of big data, but also establishes a complete security protection system and realizes the security management of the entire system.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are Some embodiments of the present invention may also be used to obtain other drawings based on these drawings without departing from the art.

The following drawings are only intended to illustrate and explain the present invention, and do not limit the scope of the invention.

FIG. 1 is a schematic structural diagram of a big data management system according to an embodiment of the present invention.

FIG. 2 is a schematic structural diagram of another big data management system according to an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of a cloud storage encryption/decryption unit according to an embodiment of the present invention.

Embodiments of the invention

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

As shown in FIG. 1 , in the embodiment of the present invention, a big data management system includes: a data collection unit 100, a data parallel processing unit 200, a cloud storage unit 300, a cloud storage encryption/decryption unit 400, and a data access authority authentication unit. 500. The data redundancy determining unit 600 and the data deduplication unit 700. The cloud storage unit 300 includes a local storage 310 and a cloud storage 320. The cloud storage encryption/decryption unit 400 is respectively associated with the local storage 310 and the cloud storage 320. Connected to the data access authority authentication unit 700; the data redundancy determination unit 600 is connected to the data collection unit 100, the local storage 310, and the cloud storage 320, respectively; the data repetition deletion unit 700 is respectively associated with the The local storage 310, the cloud storage 320, and the data redundancy judging unit 600 are connected. Meanwhile, the system further includes: one or more clients 800, the client 800 is connected to the cloud storage encryption and decryption unit 400, and by accessing the cloud storage unit, determining whether there is permission to access the target data point and acquiring the target data. The location information of the cloud storage unit to which the point belongs or directly access the location letter The cloud storage unit corresponding to the information; and the location information of the cloud storage unit to which the accessed target data point belongs. It not only improves the processing efficiency and speed of big data, but also establishes a complete security protection system and realizes the security management of the entire system.

In the embodiment of the present invention, the data parallel processing unit 200 adopts a parallel data preprocessing technology, and has a Map/Reduce processing model. Each of the processing tasks is processed in parallel by multiple Map tasks by calling a Map function. It is assigned to the execution node assigned to the processing task, and then the result of the processing of each Map task of each processing task is merged by calling the Reduce function to complete the data preprocessing.

In the embodiment of the present invention, the cloud storage unit 300 adopts a discrete distributed data storage end. The discrete distributed data storage end may be an HBase structural database and an HDFS structural database; or the discrete distributed data storage end may be an MPP relational database and a Hadoop structural database.

In the embodiment of the present invention, as shown in FIG. 2, the system further includes: a data integration unit 900, configured to integrate discrete source data, and store the integrated source data to the cloud storage unit.

In the embodiment of the present invention, as shown in FIG. 3, the cloud storage encryption/decryption unit 400 includes a cloud storage security encryption chip 410 and a cloud storage security decryption chip 420. The security encryption chip 410 and the security decryption chip 420 are used. Secure Sockets Layer SSL and/or Secure Transport Layer TLS. The security encryption chip 410 and the security decryption chip 420 include one or more security processing units of AES, RSA, SHA, OTP, RNG, and GUID.

In the embodiment of the present invention, the data access authority authentication unit 500 may perform encryption and decryption authentication on the data by using an RSA method.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. Within the scope.

Claims

A big data management system, comprising: a data collection unit, a data parallel processing unit, a cloud storage unit, a cloud storage encryption and decryption unit, a data access authority authentication unit, a data redundancy determination unit, and a data deduplication unit; The cloud storage unit includes a local storage and a cloud storage; the cloud storage encryption and decryption unit is respectively connected to the local storage, the cloud storage, and the data access authority authentication unit; and the data redundancy determination unit and the data collection unit respectively And the local storage and the cloud storage connection; the data deduplication unit is respectively connected to the local storage, the cloud storage, and the data redundancy judging unit.
The big data management system of claim 1 further comprising:

One or more clients, the client is connected to the cloud storage encryption and decryption unit, and by accessing the cloud storage unit, determining whether there is permission to access the target data point and obtaining the location information of the cloud storage unit to which the target data point belongs or directly Visit the location letter The cloud storage unit corresponding to the information; and the location information of the cloud storage unit to which the accessed target data point belongs.
The big data management system according to claim 1, wherein the data parallel processing unit adopts a parallel data preprocessing technology, and has a Map/Reduce processing model, and each processing task is represented by multiple maps by calling a Map function. The tasks are processed in parallel. These Map tasks are assigned to the execution nodes assigned to the processing tasks, and then the Reduce function is used to merge the processing results of each Map task of each processing task to complete the data preprocessing.
The big data management system of claim 1 wherein said cloud storage unit employs a discrete distributed data storage.
The big data management system according to claim 4, wherein said discrete distributed data storage end comprises an HBase structured database and an HDFS structured database.
The big data management system according to claim 4, wherein said discrete distributed data storage comprises: an MPP relational database and a Hadoop structure database.
The big data management system of claim 1 further comprising:

A data integration unit for integrating discrete source data and storing the consolidated source data to a cloud storage unit.
The big data management system according to claim 1, wherein the cloud storage encryption/decryption unit comprises a cloud storage security encryption chip and a cloud storage security decryption chip, and the security encryption chip and the security decryption chip adopt a secure socket. Layer SSL and/or Secure Transport Layer TLS.
The big data management system according to claim 8, wherein the secure encryption chip and the secure decryption chip comprise one or more security processing units of AES, RSA, SHA, OTP, RNG, GUID.
The big data management system according to claim 1, wherein the data access authority authentication unit performs encryption and decryption authentication on the data by using an RSA method.