CN105488235A - Cloud platform data management system based on industrial big data and construction method thereof - Google Patents

Cloud platform data management system based on industrial big data and construction method thereof Download PDF

Info

Publication number
CN105488235A
CN105488235A CN201610079827.5A CN201610079827A CN105488235A CN 105488235 A CN105488235 A CN 105488235A CN 201610079827 A CN201610079827 A CN 201610079827A CN 105488235 A CN105488235 A CN 105488235A
Authority
CN
China
Prior art keywords
data
module
management system
cloud platform
aggregate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610079827.5A
Other languages
Chinese (zh)
Inventor
赵俊涛
李鹏飞
胡尊亭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Jianwei Iot Technology Co Ltd
Original Assignee
Suzhou Jianwei Iot Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Jianwei Iot Technology Co Ltd filed Critical Suzhou Jianwei Iot Technology Co Ltd
Priority to CN201610079827.5A priority Critical patent/CN105488235A/en
Publication of CN105488235A publication Critical patent/CN105488235A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/80Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a cloud platform data management system based on industrial big data. The system comprises a data acquisition system, an industrial field data module, a Hadoop cluster module, a data aggregation module, a data distribution module and a data persistent storage module. The industrial field data module is located in the data acquisition system and connected with the Hadoop cluster module, the Hadoop cluster module is connected with the data aggregation module and sends processed data to the data aggregation module, the data aggregation module is connected with a data analysis module, and the data aggregation module sends the processed data to the data analysis module to be analyzed. The data distribution module is connected with the data persistent storage module, and the data distribution distributes received data into the data persistent storage module. By means of the system, the size of data blocks can be reduced, data storage efficiency is improved, and safety and reliability of data are guaranteed.

Description

A kind of cloud platform data management system based on the large data of industry and construction method thereof
Technical field
The present invention relates to a kind of data management system, particularly relate to a kind of data management system based on the large data of industry and construction method.
Background technology
Along with the continuous maturation of cloud computing technology, the features such as cloud computing is virtual with it, highly reliable, easily extensible, low cost are widely used, increasing enterprise by cloud computing technology just its data center be stored to high in the clouds, thus ensure the reliability of data, and save great amount of cost.Cloud platform architecture mainly comprises three layers from bottom to up, namely namely infrastructure serve (IaaS), platform namely serves (PaaS) and namely software serve (SaaS), the cloud computing correlation technique of current comparative maturity mainly contains openstack, Hadoop, spark etc., and the cloud platform data management system of main flow is also all build on their basis.At present based on the cloud platform data management system constructing plan comparative maturity of this application scenarios of consumer level mass data, and widely applied, compared with consumer level data, industrial data is to the real-time of platform, reliability and safety and reliability have higher requirement, therefore existing cloud platform data management system constructing plan can not well be applied in large this scene of data of industry, and the special cloud platform data management system constructing plan for the large data of industry is also fewer at present, therefore how according to the feature of industrial data itself, invent a kind of cloud platform data management system constructing plan that can adapt to industrial requirement, it is a current more urgent problem.
Current existing cloud platform data management system constructing plan is all based on this application scenarios of consumer level mass data, and industrial data cloud platform management system has higher requirement to the real-time of data, reliability and safety, current existing cloud platform data management system constructing plan can not well be applied in industrial data environment, the present invention, according to the feature of industrial data itself, invents a kind of cloud platform data management system constructing plan that can adapt to industrial requirement.
Summary of the invention
Fundamental purpose of the present invention is the feature according to the large data of industry itself, invent a kind of cloud platform data management system constructing plan that can adapt to industrial requirement, the requirement of the large data of industry to cloud platform data management system reliability, real-time and security can be met.
For solving the problem, the present invention proposes a kind of cloud platform data management system based on the large data of industry, it is characterized in that: described cloud platform data management system comprises data acquisition system (DAS), industrial field data module, Hadoop cluster module, data aggregate module, Data dissemination module and lasting data memory module, wherein, described industrial field data module is arranged in data acquisition system (DAS), the destructuring industrial data collected is transferred to industrial field data module by described data acquisition system (DAS), described industrial field data module is connected with described Hadoop cluster module, described Hadoop cluster module and described data aggregate model calling, data after process are sent to described data aggregate module by described Hadoop cluster module, described data aggregate module is connected with described data analysis module, and the data after process send to described data analysis module to analyze by described data aggregate module, described Data dissemination module is connected with described lasting data memory module, and the data received are assigned in lasting data memory module by described Data dissemination module.
Preferably, described cloud platform data management system also comprises security module, and described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively.
Preferably, described Hadoop cluster module comprises data structured module, data debug module, data deduplication module, Data Integration module, is connected in series between each module.
Preferably, described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module and described data coupling model calling, described data coupling module is connected with described data compressing module.
The invention also discloses a kind of construction method according to the above-mentioned cloud platform data management system based on the large data of industry, comprise the following steps:
S1. described data acquisition system industrial field data, and the data collected are transferred to industrial field data module;
The destructuring industrial data received is carried out structuring process and generates semi-structured data by S2. described industrial field data module, then gives described Hadoop cluster module by described semi-structured data by Internet Transmission;
S3. the described semi-structured data described in the process of Hadoop cluster module, and send to described data aggregate module;
S4. the described corresponding data of data aggregate resume module, and send to described Data dissemination module;
The data received are assigned in lasting data memory module by S5. described Data dissemination module.
Preferably, described construction method also comprises step S6: described cloud platform data management system also comprises security module, and described security module communicates with each processing module, ensures the security of data.
Preferably, step S3 comprises: described Hadoop cluster module also comprises data structured module, data debug module, data deduplication module, Data Integration module, the described semi-structured data that described data structured modular structure process receives, generating structured data; Described data debug module removes the structural data of mistake, described data deduplication module row except the structural data repeated, the data after the module integrated described data debug module of described Data Integration and the process of described data deduplication module.
Preferably, step S4 comprises, and described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module by similar data gathering together, generates set of metadata of similar data class, and the module integrated described set of metadata of similar data class of described data coupling, optimizes the data of described set of metadata of similar data class; Described data compressing module calling data compression algorithm compresses the data of described set of metadata of similar data class.
Technical scheme of the present invention has following beneficial effect:
(1) distributed thought is utilized, the structuring process of situ industrial data is transferred to enterprises end, data after such process directly can consign to Hadoop cluster and use, and not only reduce the load of cloud platform data management system, and improve the real-time of cloud platform data process.
(2) after data clusters is generated set of metadata of similar data class, the basis of Data Integration is before integrated set of metadata of similar data class again, further increases the validity of data.
(3) in Data dissemination module, usage data compression algorithm is compressed a set of metadata of similar data block, thus reduces the size of data block, improves data storage efficiency.
(4) in data processing and transmitting procedure, use safety module carries out repeatedly authentication and authorization, fully ensures the reliability of data.
Accompanying drawing explanation
Fig. 1 is the schematic diagram of a kind of cloud platform data management system based on the large data of industry of the present invention.
Embodiment
For making the object of the embodiment of the present invention, technical scheme and advantage clearly, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is the present invention's part embodiment, instead of whole embodiments.
As the schematic diagram that Fig. 1 is a kind of cloud platform data management system based on the large data of industry of the present invention.Wherein, described cloud platform data management system comprises data acquisition system (DAS), industrial field data module, Hadoop cluster module, data aggregate module, Data dissemination module and lasting data memory module, wherein, described industrial field data module is arranged in data acquisition system (DAS), the destructuring industrial data collected is transferred to industrial field data module by described data acquisition system (DAS), described industrial field data module is connected with described Hadoop cluster module, described Hadoop cluster module and described data aggregate model calling, data after process are sent to described data aggregate module by described Hadoop cluster module, described data aggregate module is connected with described data analysis module, and the data after process send to described data analysis module to analyze by described data aggregate module, described Data dissemination module is connected with described lasting data memory module, and the data received are assigned in lasting data memory module by described Data dissemination module.
Described cloud platform data management system also comprises security module, and described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively.
Described Hadoop cluster module comprises data structured module, data debug module, data deduplication module, Data Integration module, is connected in series between each module.
Described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module and described data coupling model calling, described data coupling module is connected with described data compressing module.
The data received are assigned in lasting data memory module by described Data dissemination module, ensure the reliability of data.Data dissemination inside modules safeguards a list, current each storage subsystem information effective of this list records, when selecting storage subsystem, Data dissemination module uses in hash algorithm from then on list and selects a storage subsystem, the set of metadata of similar data class obtained is stored in this subsystem in data aggregate module.In addition, when selecting storage subsystem, the availability of subsystems, active volume, network condition etc. to also be considered.
Described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively, communicate with each processing module, ensure the security of data, especially the security of data in data processing and transmitting procedure is ensured, all to communicate with security module in each process of data processing and transmission, after only having the authentication and authorization by security module, just can carry out next step process, thus ensure the security of data.
The invention also discloses a kind of construction method according to the above-mentioned cloud platform data management system based on the large data of industry, comprise the following steps:
S1. described data acquisition system industrial field data, and the data collected are transferred to industrial field data module;
The destructuring industrial data received is carried out structuring process and generates semi-structured data by S2. described industrial field data module, then gives described Hadoop cluster module by described semi-structured data by Internet Transmission;
S3. the described semi-structured data described in the process of Hadoop cluster module, and send to described data aggregate module;
S4. the described corresponding data of data aggregate resume module, and send to described Data dissemination module;
The data received are assigned in lasting data memory module by S5. described Data dissemination module.
Described construction method also comprises step S6: described cloud platform data management system also comprises security module, and described security module communicates with each processing module, ensures the security of data.
Described Hadoop cluster module also comprises data structured module, data debug module, data deduplication module, Data Integration module, the described semi-structured data that described data structured modular structure process receives, generating structured data; Described data debug module removes the structural data of mistake, described data deduplication module row except the structural data repeated, the data after the module integrated described data debug module of described Data Integration and the process of described data deduplication module.
Described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module by similar data gathering together, generates set of metadata of similar data class, and the module integrated described set of metadata of similar data class of described data coupling, optimizes the data of described set of metadata of similar data class, ensure the validity of data; Described data compressing module calling data compression algorithm compresses the data of described set of metadata of similar data class, reduces size of data, finally data is consigned to Data dissemination module.
Finally, by carrying out structuring process at enterprises end to data, data directly can be consigned to Hadoop cluster, ensure that the real-time of system, store by data aggregate, Data dissemination, lasting data validity, the reliability that these three modules ensure that data.
Based on the embodiment in the present invention, those of ordinary skill in the art, not making other embodiments all obtained under creative work prerequisite, belong to the scope of protection of the invention.Although the present invention illustrates with regard to preferred implementation and describes, only it will be understood by those of skill in the art that otherwise exceed claim limited range of the present invention, variations and modifications can be carried out to the present invention.

Claims (8)

1. the cloud platform data management system based on the large data of industry, it is characterized in that: described cloud platform data management system comprises data acquisition system (DAS), industrial field data module, Hadoop cluster module, data aggregate module, Data dissemination module and lasting data memory module, wherein, described industrial field data module is arranged in data acquisition system (DAS), the destructuring industrial data collected is transferred to industrial field data module by described data acquisition system (DAS), described industrial field data module is connected with described Hadoop cluster module, described Hadoop cluster module and described data aggregate model calling, data after process are sent to described data aggregate module by described Hadoop cluster module, described data aggregate module is connected with described data analysis module, and the data after process send to described data analysis module to analyze by described data aggregate module, described Data dissemination module is connected with described lasting data memory module, and the data received are assigned in lasting data memory module by described Data dissemination module.
2. a kind of cloud platform data management system based on the large data of industry according to claim 1, it is characterized in that: described cloud platform data management system also comprises security module, described security module is connected with described Hadoop cluster module, described data aggregate module, described Data dissemination module, described lasting data memory module respectively.
3. a kind of cloud platform data management system based on the large data of industry according to claim 1 and 2, it is characterized in that: described Hadoop cluster module comprises data structured module, data debug module, data deduplication module, Data Integration module, is connected in series between each module.
4. a kind of cloud platform data management system based on the large data of industry according to claim 1 and 2, is characterized in that: described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module and described data coupling model calling, described data coupling module is connected with described data compressing module.
5., according to a construction method for a kind of cloud platform data management system based on the large data of industry of the claims 1-4, comprise the following steps:
S1. described data acquisition system industrial field data, and the data collected are transferred to industrial field data module;
The destructuring industrial data received is carried out structuring process and generates semi-structured data by S2. described industrial field data module, then gives described Hadoop cluster module by described semi-structured data by Internet Transmission;
S3. the described semi-structured data described in the process of Hadoop cluster module, and send to described data aggregate module;
S4. the described corresponding data of data aggregate resume module, and send to described Data dissemination module;
The data received are assigned in lasting data memory module by S5. described Data dissemination module.
6. the construction method of a kind of cloud platform data management system based on the large data of industry according to claim 5, it is characterized in that: described construction method also comprises step S6: described cloud platform data management system also comprises security module, described security module communicates with each processing module, ensures the security of data.
7. the construction method of a kind of cloud platform data management system based on the large data of industry according to claim 5, it is characterized in that: described step S3 comprises, described Hadoop cluster module also comprises data structured module, data debug module, data deduplication module, Data Integration module, first, the described semi-structured data that described data structured modular structure process receives, generating structured data; Described data debug module removes the structural data of mistake, described data deduplication module row except the structural data repeated, the data after the module integrated described data debug module of described Data Integration and the process of described data deduplication module.
8. the construction method of a kind of cloud platform data management system based on the large data of industry according to claim 5, it is characterized in that: described step S4 comprises, described data aggregate module comprises data clusters module, data coupling module and data compressing module; Described data clusters module by similar data gathering together, generates set of metadata of similar data class, and the module integrated described set of metadata of similar data class of described data coupling, optimizes the data of described set of metadata of similar data class; Described data compressing module calling data compression algorithm compresses the data of described set of metadata of similar data class.
CN201610079827.5A 2016-02-03 2016-02-03 Cloud platform data management system based on industrial big data and construction method thereof Pending CN105488235A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610079827.5A CN105488235A (en) 2016-02-03 2016-02-03 Cloud platform data management system based on industrial big data and construction method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610079827.5A CN105488235A (en) 2016-02-03 2016-02-03 Cloud platform data management system based on industrial big data and construction method thereof

Publications (1)

Publication Number Publication Date
CN105488235A true CN105488235A (en) 2016-04-13

Family

ID=55675210

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610079827.5A Pending CN105488235A (en) 2016-02-03 2016-02-03 Cloud platform data management system based on industrial big data and construction method thereof

Country Status (1)

Country Link
CN (1) CN105488235A (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106885604A (en) * 2017-02-15 2017-06-23 重庆工商职业学院 A kind of industrial feeding vehicle based on big data feeds intake quantity collection system in real time
CN107066551A (en) * 2017-03-23 2017-08-18 中国科学院计算技术研究所 The line and column storage method and system of a kind of tree shaped data
CN107315769A (en) * 2017-05-18 2017-11-03 北京安点科技有限责任公司 Simplify and processing system with reference to the mass data of multifactor optimization technology and MapReduce technologies
CN107480244A (en) * 2017-08-10 2017-12-15 成都天衡电科科技有限公司 A kind of industrial data collects and processing system and its processing method
CN108021051A (en) * 2016-10-31 2018-05-11 无锡云汇科技有限公司 Industrial control unit (ICU)
CN109460498A (en) * 2018-11-07 2019-03-12 广州小天软件有限公司 A kind of verification of data method and device
CN110304510A (en) * 2019-05-31 2019-10-08 安徽电梯大叔科技有限公司 A kind of intelligent elevator supervisory systems
CN112699108A (en) * 2020-12-25 2021-04-23 中科恒运股份有限公司 Data reconstruction method and device for marital registration system and terminal equipment

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102566493A (en) * 2012-01-17 2012-07-11 上海交通大学 Data acquiring and processing embedded adapter of numerical control machine
CN104156810A (en) * 2014-07-31 2014-11-19 国网山东省电力公司 Power dispatching production management system based on cloud computing and realization method of power dispatching production management system
CN104410662A (en) * 2014-10-23 2015-03-11 山东大学 Parallel mass data transmitting middleware of Internet of things and working method thereof
US20150095384A1 (en) * 2013-09-27 2015-04-02 Tata Consultancy Services Limited File transfer to a distributed file system
CN104850640A (en) * 2015-05-26 2015-08-19 华北电力大学(保定) HBase based storage and query method and system for power equipment status monitoring data
CN105045856A (en) * 2015-07-09 2015-11-11 中国资源卫星应用中心 Hadoop-based data processing system for big-data remote sensing satellite
CN105260448A (en) * 2015-10-10 2016-01-20 成都博元时代软件有限公司 Big data information analysis method

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102566493A (en) * 2012-01-17 2012-07-11 上海交通大学 Data acquiring and processing embedded adapter of numerical control machine
US20150095384A1 (en) * 2013-09-27 2015-04-02 Tata Consultancy Services Limited File transfer to a distributed file system
CN104156810A (en) * 2014-07-31 2014-11-19 国网山东省电力公司 Power dispatching production management system based on cloud computing and realization method of power dispatching production management system
CN104410662A (en) * 2014-10-23 2015-03-11 山东大学 Parallel mass data transmitting middleware of Internet of things and working method thereof
CN104850640A (en) * 2015-05-26 2015-08-19 华北电力大学(保定) HBase based storage and query method and system for power equipment status monitoring data
CN105045856A (en) * 2015-07-09 2015-11-11 中国资源卫星应用中心 Hadoop-based data processing system for big-data remote sensing satellite
CN105260448A (en) * 2015-10-10 2016-01-20 成都博元时代软件有限公司 Big data information analysis method

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108021051A (en) * 2016-10-31 2018-05-11 无锡云汇科技有限公司 Industrial control unit (ICU)
CN106885604A (en) * 2017-02-15 2017-06-23 重庆工商职业学院 A kind of industrial feeding vehicle based on big data feeds intake quantity collection system in real time
CN106885604B (en) * 2017-02-15 2018-12-14 重庆工商职业学院 A kind of industrial feeding vehicle based on big data feeds intake quantity collection system in real time
CN107066551B (en) * 2017-03-23 2020-04-03 中国科学院计算技术研究所 Row-type and column-type storage method and system for tree-shaped data
CN107066551A (en) * 2017-03-23 2017-08-18 中国科学院计算技术研究所 The line and column storage method and system of a kind of tree shaped data
CN107315769A (en) * 2017-05-18 2017-11-03 北京安点科技有限责任公司 Simplify and processing system with reference to the mass data of multifactor optimization technology and MapReduce technologies
CN107315769B (en) * 2017-05-18 2021-03-12 北京安点科技有限责任公司 Mass data simplifying and processing system combining multi-factor analysis technology and MapReduce technology
CN107480244A (en) * 2017-08-10 2017-12-15 成都天衡电科科技有限公司 A kind of industrial data collects and processing system and its processing method
CN113220776A (en) * 2017-08-10 2021-08-06 成都天衡智造科技有限公司 Industrial data processing system and method
CN113220776B (en) * 2017-08-10 2022-06-17 成都天衡智造科技有限公司 Industrial data processing system and method
CN109460498A (en) * 2018-11-07 2019-03-12 广州小天软件有限公司 A kind of verification of data method and device
CN110304510A (en) * 2019-05-31 2019-10-08 安徽电梯大叔科技有限公司 A kind of intelligent elevator supervisory systems
CN112699108A (en) * 2020-12-25 2021-04-23 中科恒运股份有限公司 Data reconstruction method and device for marital registration system and terminal equipment

Similar Documents

Publication Publication Date Title
CN105488235A (en) Cloud platform data management system based on industrial big data and construction method thereof
CN109213600B (en) GPU resource scheduling method and device based on AI cloud
CN104331421A (en) High-efficiency processing method and system for big data
CN103873438A (en) Compression packet uploading and duplication-removing system and method
CN112270833B (en) Trajectory fitting method and device, electronic equipment and storage medium
CN111586091B (en) Edge computing gateway system for realizing computing power assembly
CN103778034A (en) Cloud storage-based data backup disaster recovery method and system
CN105338027A (en) Method, system and device for cloud storage of video data
CN105045856A (en) Hadoop-based data processing system for big-data remote sensing satellite
CN105302920A (en) Optimal management method and system for cloud storage data
CN102750368B (en) High-speed importing method of cluster data in data base
CN103617162A (en) Method of constructing Hilbert R-tree index on equivalent cloud platform
CN107612984B (en) Big data platform based on internet
CN113568938B (en) Data stream processing method and device, electronic equipment and storage medium
CN103986783A (en) Cloud computing system
CN103823807A (en) Data de-duplication method, device and system
CN105282045B (en) A kind of distributed computing and storage method based on consistency hash algorithm
CN205540723U (en) Information retrieval system based on cloud calculates
CN108306965A (en) The data processing method and device of camera, storage medium, camera
CN105049485A (en) Real-time video processing oriented load-aware cloud calculation system
CN108959614A (en) A kind of snapshot management method, system, device, equipment and readable storage medium storing program for executing
CN204906437U (en) Big data storage application network framework
CN205899536U (en) Geographic information service system based on tile map
CN111241069A (en) Data flattening method and system based on block chain
CN203166994U (en) Data server based on cloud computing

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20160413

WD01 Invention patent application deemed withdrawn after publication