CN108196797B - Data processing system based on cloud computing - Google Patents
Data processing system based on cloud computing Download PDFInfo
- Publication number
- CN108196797B CN108196797B CN201810076931.8A CN201810076931A CN108196797B CN 108196797 B CN108196797 B CN 108196797B CN 201810076931 A CN201810076931 A CN 201810076931A CN 108196797 B CN108196797 B CN 108196797B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- storage
- cloud computing
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0625—Power saving in storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0653—Monitoring storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing system based on cloud computing, which comprises a data acquisition unit for acquiring a large-scale sensing source data stream, a cloud computing center for receiving, processing and storing data of the data acquisition unit, a user terminal, an administrator terminal and a backup server for data backup. According to the data processing method and system, the data processing server is used for processing the acquired data, the data which is processed by the node is filtered, the data processing overhead is reduced, meanwhile, the data processing is localized, the data transmission overhead among the nodes is reduced, the storage energy consumption is reduced, three-layer data storage of the acquired data is performed by using the database server aiming at the characteristics of heterogeneity, uncertainty, high data flow and the like of a large amount of data acquired in cloud computing, the expandability and fault tolerance of the data storage are improved, and the data processing system is suitable for the data acquisition requirements of different users, has strong applicability and wide application range.
Description
Technical Field
The invention belongs to the technical field of data processing systems, and particularly relates to a data processing system based on cloud computing.
Background
With the development of computer technology and society, all the world is in a huge data group. The traditional data processing mode is to use a mainframe with strong processing capability and a relational database, the storage mode is to store all the data on the mainframe with large disk capacity, although the storage capacity of the disk is increasing in recent years, the access speed of the data is not advanced, so various database manufacturers have to develop a new generation of database to meet the challenge of large data. In consideration of the problems of capacity, parallel processing and the like, experts propose to establish a distributed database by using a distributed technology and store mass data by using a cluster mode, and the cloud computing technology is well applied.
The data processed in a cloud computing environment is different from the traditional data, and a large amount of data acquired in the cloud computing has the characteristics of heterogeneity, uncertainty, high data flow and the like.
Disclosure of Invention
The invention provides a data processing system based on cloud computing, which can solve the problems.
A data processing system based on cloud computing comprises a data acquisition unit for acquiring a large-scale sensing source data stream, a cloud computing center for receiving, processing and storing data of the data acquisition unit, a user terminal, an administrator terminal and a backup server for data backup;
the data acquisition unit comprises a sensor node, a radio frequency identifier and a camera for collecting video data streams;
the cloud computing center comprises a central control server, a data processing server for processing the data acquired by the data acquisition unit, a database server for storing the data processed by the data processing server, a data management server for managing the data stored by the database server and an application interface;
the application interface comprises a Bluetooth communication module, a Wi-Fi communication module and a network communication module;
when the data processing server processes data, preprocessed data are distributed and cached in each node under a Hadoop cloud computing framework, each node receives data flow redundantly, the nodes filter out data which are responsible for processing by the node in a Map stage, calculation in a Reduce stage is performed on a local cache, intermediate result grouping multiplexing is performed on calculation results in the Reduce stage on local storage, and finally the local calculation results are synchronized to the distributed storage area.
The database server comprises three layers of data storage units, a central storage scheduling unit, a storage state monitoring unit, a storage virtualization unit and a storage layer-management layer interface unit, wherein the three layers of data storage units comprise a first data layer used for storing and dynamically updating data collected by the data collection unit and intermediate results of each group calculated by the data processing server, a second data layer used for storing and dynamically updating final data processing results of the data processing server, and a third data layer used for stripping data required to be evolved into historical data from the second data layer so as to store and update the historical data, the central storage scheduling unit schedules data sets in the three layers of data storage units respectively according to instructions of the central control server and keeps data consistency of the first data layer and the second data layer in the scheduling process, the storage state monitoring unit monitors the storage state of data, the storage virtualization unit realizes autonomous dynamic allocation and scheduling of computing resources of the data mining cloud service by using a virtualization technology, and the storage layer-management layer interface unit realizes data interaction between the database server and the data management server;
the cloud computing center is respectively connected with the data acquisition unit, the user terminal, the administrator terminal and the backup server;
and the central control server in the cloud computing center is respectively connected with the application interface, the data processing server, the database server and the data management server.
Preferably, the data acquisition unit is connected to the data processing server through a wireless network; the backup server is respectively connected with the database server and the data management server; the administrator terminal is connected with the central control server; and the user terminal is connected with the central control server through the application interface.
Preferably, the data processing system further comprises a backup memory for storing data of the backup server, and the backup memory is connected to the backup server.
Preferably, the user terminal and the administrator terminal are a notebook, a mobile phone and a tablet computer.
Preferably, the backup memory is a mobile hard disk, a computer, a U disk or other storage devices.
The invention has the beneficial effects that: the data processing system based on cloud computing provided by the invention filters the data which is processed by the node through the processing of the data processing server, reduces the repeated processing overhead of the existing historical data when the acquired data reaches the data processing server each time, localizes the data processing, reduces the data transmission overhead among the nodes, reduces the storage energy consumption, utilizes the database server to store the three-layer data of the acquired data aiming at the characteristics of heterogeneity, uncertainty, high data flow and the like of a large amount of data acquired in the cloud computing, improves the expandability and fault tolerance of the data storage, utilizes the backup server to backup the system data, prevents the important data of the data processing system from being lost, ensures the recoverability of the data processing system, and is suitable for the data acquisition requirements of different users, has strong applicability and wide application range.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic overall structure diagram of a data processing system based on cloud computing according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a data storage structure of a data processing system based on cloud computing according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a real-time data processing structure according to an embodiment of the present invention.
Description of reference numerals:
1-data acquisition unit, 2-central control server, 3-data processing server, 4-database server, 5-data management server, 6-user terminal, 7-administrator terminal, 8-backup server, 9-backup memory, 10-application interface.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1 to 3, a data processing system based on cloud computing includes a data acquisition unit 1 for acquiring a large-scale perceptual source data stream, a cloud computing center for receiving, processing and storing data of the data acquisition unit 1, a user terminal 6, an administrator terminal 7, and a backup server 8 for data backup;
the data acquisition unit 1 comprises a sensor node, a radio frequency identifier and a camera for collecting video data streams;
the cloud computing center comprises a central control server 2, a data processing server 3 for processing data acquired by the data acquisition unit 1, a database server 4 for storing the data processed by the data processing server 3, a data management server 5 for managing the data stored by the database server 4 and an application interface 10;
the application interface 10 comprises a Bluetooth communication module, a Wi-Fi communication module and a network communication module;
when the data processing server 3 processes data, preprocessed data are distributed and cached in each node under a Hadoop cloud computing framework, each node redundantly receives data streams, the nodes filter out data which are responsible for processing by the node in a Map stage, calculation in a Reduce stage is performed on a local cache, intermediate result grouping multiplexing is performed on calculation results in the Reduce stage on local storage, and finally the local calculation results are synchronized to the distributed storage area.
The database server 4 comprises three layers of data storage units, a central storage scheduling unit, a storage state monitoring unit, a storage virtualization unit and a storage layer-management layer interface unit, wherein the three layers of data storage units comprise a first data layer for storing and dynamically updating the data collected by the data collection unit 1 and the intermediate result of each group calculated by the data processing server 3, a second data layer for storing and dynamically updating the final data processing result of the data processing server 3, and a third data layer for stripping the data required to be evolved into historical data from the second data layer so as to store and update the historical data, the central storage scheduling unit schedules data sets in the three layers of data storage units respectively according to the instruction of the central control server 2 and keeps the data consistency of the first data layer and the second data layer in the scheduling process, the storage state monitoring unit monitors the storage state of data, the storage virtualization unit realizes autonomous dynamic allocation and scheduling of computing resources of the data mining cloud service by using a virtualization technology, and the storage layer-management layer interface unit realizes data interaction between the database server 4 and the data management server 5;
the cloud computing center is respectively connected with the data acquisition unit 1, the user terminal 6, the administrator terminal 7 and the backup server 8;
the central control server 2 in the cloud computing center is respectively connected with the application interface 10, the data processing server 3, the database server 4 and the data management server 5.
Preferably, the data acquisition unit 1 is connected to the data processing server 3 through a wireless network; the backup server 8 is respectively connected with the database server 4 and the data management server 5; the administrator terminal 7 is connected with the central control server 2; the user terminal 6 is connected with the central control server 2 through the application interface 10.
Preferably, the data processing system further comprises a backup memory 9 for storing data of the backup server 8, and the backup memory 9 is connected to the backup server 8.
Preferably, the user terminal 6 and the administrator terminal 7 are a notebook, a mobile phone, and a tablet computer.
Preferably, the backup memory 9 is a mobile hard disk, a computer, a usb disk or other storage devices.
When the data backup system is used, the data acquisition unit 1 acquires data and stores the data to the cloud computing center, the data processing server 3 processes the data acquired by the data acquisition unit 1, the database server 4 utilizes the three-layer data storage units to store the data processed by the data processing server 3 in a layered manner, the data management server 5 is utilized to uniformly manage the data stored by the database server 4, when the central control server 2 receives an operation instruction sent by the administrator terminal 7, the central control server 2 calls data resources stored in the database server 4 and sends data requested by a user to different user terminals 6 through the application interface 10, and the backup server 8 backs up the data stored in the database server 4 to prevent data loss, the safety of data storage is improved, and when data loss occurs, the data backed up by the backup server 8 is transmitted to the database server 4 through the data management server 5 for data storage.
According to the data processing system based on cloud computing, the data which is processed by the node is filtered through the processing of the data processing server 3, the repeated processing overhead of existing historical data when the acquired data reaches the data processing server 3 is reduced, the data processing is localized, the data transmission overhead among the nodes is reduced, the storage energy consumption is reduced, the three-layer data storage of the acquired data is carried out by using the database server 4 aiming at the characteristics of heterogeneity, uncertainty, high data flow and the like of a large amount of data acquired in the cloud computing, the expandability and fault tolerance of the data storage are improved, the backup server 8 is used for carrying out backup processing on the system data, the important data of the data processing system is prevented from being lost, the recoverability of the data processing system is ensured, and the data processing system is suitable for the data acquisition requirements of different users, has strong applicability and wide application range.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (4)
1. A data processing system based on cloud computing is characterized by comprising a data acquisition unit (1) for acquiring a large-scale perception source data stream, a cloud computing center for receiving, processing and storing data of the data acquisition unit (1), a user terminal (6), an administrator terminal (7) and a backup server (8) for data backup;
the data acquisition unit (1) comprises a sensor node, a radio frequency identifier and a camera for collecting video data streams;
the cloud computing center comprises a central control server (2), a data processing server (3) used for processing data acquired by the data acquisition unit (1), a database server (4) used for storing the data processed by the data processing server (3), a data management server (5) used for managing the data stored by the database server (4) and an application interface (10);
the application interface (10) comprises a Bluetooth communication module and a network communication module;
when the data processing server (3) processes data, the preprocessed data are distributed and cached in each node under a Hadoop cloud computing framework, each node receives data streams redundantly, the nodes filter the data which are processed by the node in a Map stage, the nodes perform calculation in a Reduce stage on a local cache, intermediate result grouping multiplexing is performed on the calculation result in the Reduce stage on a local storage, and finally the local calculation result is synchronized to a distributed storage area;
the database server (4) comprises three layers of data storage units, a central storage scheduling unit, a storage state monitoring unit, a storage virtualization unit and a storage layer-management layer interface unit, wherein the three layers of data storage units comprise a first data layer, a second data layer and a third data layer, the first data layer is used for storing and dynamically updating data collected by the data collection unit (1) and intermediate results of all groups calculated by the data processing server (3), the second data layer is used for storing and dynamically updating final data processing results of the data processing server (3), the third data layer is used for stripping data needing to be evolved into historical data from the second data layer so as to store and update the historical data, the central storage scheduling unit schedules data sets in the three layers of data storage units respectively according to instructions of the central control server (2), the data consistency of a first data layer and a second data layer in the scheduling process is kept, the storage state monitoring unit monitors the storage state of data, the storage virtualization unit realizes autonomous dynamic allocation and scheduling of computing resources of the data mining cloud service by using a virtualization technology, and the storage layer-management layer interface unit realizes data interaction between the database server (4) and the data management server (5);
the cloud computing center is respectively connected with the data acquisition unit (1), the user terminal (6), the administrator terminal (7) and the backup server (8);
and a central control server (2) in the cloud computing center is respectively connected with the application interface (10), the data processing server (3), the database server (4) and the data management server (5).
2. A cloud computing-based data processing system as claimed in claim 1, wherein said data acquisition unit (1) is connected to said data processing server (3) via a wireless network; the backup server (8) is respectively connected with the database server (4) and the data management server (5); the administrator terminal (7) is connected with the central control server (2); the user terminal (6) is connected with the central control server (2) through the application interface (10).
3. A cloud computing-based data processing system as claimed in claim 1, further comprising a backup storage (9) for storing data of said backup server (8), said backup storage (9) being connected to said backup server (8).
4. A cloud computing-based data processing system as claimed in claim 1, wherein the user terminal (6) and administrator terminal (7) are notebooks, cell phones, tablets.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810076931.8A CN108196797B (en) | 2018-01-26 | 2018-01-26 | Data processing system based on cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810076931.8A CN108196797B (en) | 2018-01-26 | 2018-01-26 | Data processing system based on cloud computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108196797A CN108196797A (en) | 2018-06-22 |
| CN108196797B true CN108196797B (en) | 2021-01-05 |
Family
ID=62590877
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810076931.8A Expired - Fee Related CN108196797B (en) | 2018-01-26 | 2018-01-26 | Data processing system based on cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108196797B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112699101B (en) * | 2021-01-18 | 2021-09-28 | 深圳市至简科技设计有限公司 | Server system based on storage and processing |
| CN115001895A (en) * | 2022-05-25 | 2022-09-02 | 西安微电子技术研究所 | Data sharing device, system and method of satellite-borne heterogeneous system based on SPACEWIRE bus |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187943A (en) * | 2006-11-20 | 2008-05-28 | 日本电气株式会社 | Automatic update system, automatic updating method, and program therefor |
| CN101714192A (en) * | 2009-11-13 | 2010-05-26 | 航天东方红卫星有限公司 | Satellite test data processing system |
| CN107193967A (en) * | 2017-05-25 | 2017-09-22 | 南开大学 | A kind of multi-source heterogeneous industry field big data handles full link solution |
| CN107404483A (en) * | 2017-07-31 | 2017-11-28 | 北京中科金马科技股份有限公司 | Data processing method, device and data collecting system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9836229B2 (en) * | 2014-11-18 | 2017-12-05 | Netapp, Inc. | N-way merge technique for updating volume metadata in a storage I/O stack |
-
2018
- 2018-01-26 CN CN201810076931.8A patent/CN108196797B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187943A (en) * | 2006-11-20 | 2008-05-28 | 日本电气株式会社 | Automatic update system, automatic updating method, and program therefor |
| CN101714192A (en) * | 2009-11-13 | 2010-05-26 | 航天东方红卫星有限公司 | Satellite test data processing system |
| CN107193967A (en) * | 2017-05-25 | 2017-09-22 | 南开大学 | A kind of multi-source heterogeneous industry field big data handles full link solution |
| CN107404483A (en) * | 2017-07-31 | 2017-11-28 | 北京中科金马科技股份有限公司 | Data processing method, device and data collecting system |
Non-Patent Citations (2)
| Title |
|---|
| 针对高速数据流的大规模数据实时处理方法;亓开元 赵卓峰 房俊 马强;《计算机学报》;20120331;第35卷(第3期);第477页-490页 * |
| 面向大规模感知数据的实时数据流处理方法及关键技术;亓开元 韩燕波 赵卓峰 马强;《计算机集成制造系统》;20130331;第19卷(第3期);第641页-653页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108196797A (en) | 2018-06-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106033476B (en) | A kind of increment type figure calculation method under distributed computation mode in cloud computing environment | |
| CN110225074B (en) | Communication message distribution system and method based on equipment address domain | |
| CN110047014B (en) | User electric quantity data restoration method based on load curve and historical electric quantity | |
| Hao et al. | Network slicing technology in a 5G wearable network | |
| CN112671840B (en) | Cross-department data sharing system and method based on block chain technology | |
| CN103780675B (en) | A kind of cloud disc file synchronous method and device | |
| CN103533058A (en) | HDFS (Hadoop distributed file system)/Hadoop storage cluster-oriented resource monitoring system and HDFS/Hadoop storage cluster-oriented resource monitoring method | |
| CN110111092B (en) | Compatible system of payment channel | |
| CN112769897A (en) | Synchronization method and device for edge calculation message, electronic equipment and storage medium | |
| CN108924007B (en) | Big data acquisition and storage system and method of communication operation information | |
| CN108196797B (en) | Data processing system based on cloud computing | |
| CN112565415A (en) | Cross-region resource management system and method based on cloud edge cooperation | |
| CN109327335A (en) | A kind of cloud monitoring solution system and method | |
| CN110784539A (en) | Data management system and method based on cloud computing | |
| CN109660421A (en) | Method, apparatus, server and the storage medium of flexible scheduling resource | |
| CN105577423A (en) | Real-time data center cluster management system | |
| CN103516734A (en) | Data processing method, device and system | |
| CN103530335A (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN105282045B (en) | A kind of distributed computing and storage method based on consistency hash algorithm | |
| CN110213359A (en) | A kind of car networking networking data delivery system and method based on D2D | |
| CN102769495B (en) | A kind of optical fiber access network equipment communication means, Apparatus and system | |
| CN105608190B (en) | Collaborative data processing method and system | |
| CN116226067A (en) | Log management method, log management device, processor and log platform | |
| CN111614702A (en) | Edge calculation method and edge calculation system | |
| CN115022351A (en) | Storage method, device and system of battery swapping data and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210105 Termination date: 20220126 |