CN106446085A - Big data management system - Google Patents
Big data management system Download PDFInfo
- Publication number
- CN106446085A CN106446085A CN201610815529.8A CN201610815529A CN106446085A CN 106446085 A CN106446085 A CN 106446085A CN 201610815529 A CN201610815529 A CN 201610815529A CN 106446085 A CN106446085 A CN 106446085A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- service
- electrically
- management system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a big data management system in the technical field of big data. The system comprises a data acquisition module; the data acquisition module is in electric output connection with a data preprocessing unit; the data preprocessing unit is in electric output connection with a data storage system; and a data service system is in electric output connection with a data result display unit. According to the system, geographical distribution and network cross-domain of data acquisition points are supported; acquisition of different data types of different business systems is supported; data can be captured in real time; data with various types, various formats and various characteristics can be securely, reliably, quickly and effectively stored; massive data is subjected to analysis exploration and mining; data modes and characteristics are searched for; information change behind the data is looked for; data information is displayed in a conceptualized, systemized, multi-dimensional, diversified and stereoscopic manner through structure analysis; and the data management requirements necessary for information data expansion development in nowadays society are met.
Description
Technical field
The present invention relates to big data technical field, specially a kind of big data management system.
Background technology
Developing rapidly with computer technology, each FIELD Data of each row is in geometry level rapid growth.These data are derived from
Various aspects, from collecting the sensor of weather condition, digital pictures, online video data, to the transaction record of shopping at network,
Global positioning system signal of mobile phone etc., along with the drastically expansion of data scale, the data volume that every profession and trade adds up is increasing,
Data type also gets more and more, and data structure becomes increasingly complex, and has surmounted the energy of traditional data management system, tupe
Power scope, traditional serial data storehouse system has been difficult in adapt to this application demand being skyrocketed through, table in production practices
The ability revealing substantially not enough it is impossible to the management meeting the big data epoch requires, for this reason, we have proposed a kind of big data management
System comes into operation, to solve the above problems.
Content of the invention
It is an object of the invention to provide a kind of big data management system, to solve the tradition proposing in above-mentioned background technology
Serial data processing mode can not meet the problem that big data management requires.
For achieving the above object, the present invention provides following technical scheme：A kind of big data management system, including data acquisition
Module, described data acquisition module electrically exports connection data pre-processing unit, the described data pre-processing unit electrically company of output
Connect data-storage system, described data-storage system electrically exports respectively and connects data categorization module data indexing units, institute
State data categorization module data indexing units and all electrically export and connect Data Integration unit, described Data Integration unit electricity respectively
Property output Connection inquiring access port data analysis system, described data analysis system electrically export connection data modeling list
Unit, described data modeling unit electrically exports connection data service system, and described data service system electrically exports connection data
Result display unit, described data service system is electrically bi-directionally connected operation management unit.
Preferably, described data-storage system includes integrated date managment system, and described integrated date managment system is respectively
Electrically it is bi-directionally connected integrated database, real-time data base and distributed file system, described integrated database electrically exports connection
Database synchronization release unit.
Preferably, database management unit, memory database and hard disc data storehouse, institute are included inside described real-time data base
State distributed file system and include source data management, access control and redundant file parsing module.
Preferably, described data analysis system includes distributed scheduling unit, and described distributed scheduling unit electrically inputs
Connect distributed computing platform, described distributed computing platform is electrically bi-directionally connected data conversion module, data aggregate list respectively
Unit, data associating unit data excavate unit.
Preferably, described data service system includes distributed data bus, and described distributed data bus is respectively electrically
It is bi-directionally connected service management system, data access service and service logic service, described data access service and service logic clothes
Business all electrically exports and connects data cache module.
Preferably, database access unit, real-time data access unit and file are included inside described data access service
System access unit, described service logic service includes commending system, intelligently parsing module and statistical report form unit.
Preferably, described operation management unit includes monitor sensor unit, data source and instrument cluster, and described monitoring passes
Sensor cell, data source and instrument cluster all electrically export and connect acquisition configuration unit, and described acquisition configuration unit electrically exports
Connect via node, described via node electrically exports connection signal amplification unit, the described signal amplification unit electrically company of output
Connect control terminal.
Compared with prior art, the invention has the beneficial effects as follows：The geographical distribution of this system supports data collection point and net
Network is cross-domain, supports the collection of the different types of data of different business systems, can in real time data be captured, detection data
Before and after state change, on the basis of meeting coherence request, can safe and reliable, fast and effectively to polymorphic type, many lattice
Formula, the data of many characteristics are stored, and by technology such as data classification polymerization data index markers, data are processed, and
Unified efficient queried access service is provided, the data of magnanimity is analyzed exploring and excavates, search pattern and the spy of data
Levy, find the information change of data behind, by analytical structure conceptualization and systematization, and the three-dimensional logarithm of various dimensions variation
It is believed that breath is shown, meet data management requirement necessary to the development of today's society information data expansion type.
Brief description
Fig. 1 is present system block diagram；
Fig. 2 is data-storage system block diagram of the present invention；
Fig. 3 is data analysis system block diagram of the present invention；
Fig. 4 is data service system block diagram of the present invention；
Fig. 5 is operation management cellular system block diagram of the present invention；
Fig. 6 is workflow diagram of the present invention.
In figure：1 data acquisition module, 2 data pre-processing unit, 3 data-storage systems, 31 integrated date managment systems,
32 integrated databases, 33 database synchronization release units, 34 real-time data bases, 35 distributed file systems, 4 data classification moulds
Block, 5 data directory units, 6 queried access ports, 7 data analysis systems, 71 distributed scheduling units, 72 Distributed Calculation moulds
Type, 73 data conversion modules, 74 data aggregate units, 75 data associating unit, 76 data mining units, 8 data modeling units,
9 data service systems, 91 distributed data buses, 92 service management systems, 93 data access service, 94 service logic services,
95 data cache module, 10 data result display units, 11 operation management units, 111 monitor sensor unit, 112 data sources,
113 instrument cluster, 114 acquisition configuration units, 115 via nodes, 116 signal amplification units, 117 control terminals, 12 data are whole
Close unit.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation description is it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of not making creative work
Embodiment, broadly falls into the scope of protection of the invention.
Refer to Fig. 1-6, the present invention provides a kind of technical scheme：A kind of big data management system, including data acquisition module
Block 1, described data acquisition module 1 electrically exports connection data pre-processing unit 2, and described data pre-processing unit 2 electrically exports
Connect data-storage system 3, described data-storage system 3 electrically exports respectively and connects data categorization module 4 data Index List
Unit 5, described data categorization module 4 data indexing units 5 all electrically export and connect Data Integration unit 12, described Data Integration
Unit 12 electrically exports Connection inquiring access port 6 data analysis system 7 respectively, and described data analysis system 7 electrically exports
Connect data modeling unit 8, described data modeling unit 8 electrically exports connection data service system 9, described data service system
9 electrical outputs connect data result display unit 10, and described data service system 9 is electrically bi-directionally connected operation management unit 11.
Wherein, described data-storage system 3 includes integrated date managment system 31,31 points of described integrated date managment system
Electrically it is not bi-directionally connected integrated database 32, real-time data base 34 and distributed file system 35, described integrated database 32 electricity
Property output connect database synchronization release unit 33, described real-time data base 34 is internal to include database management unit, interior poke
According to storehouse and hard disc data storehouse, described distributed file system 35 includes source data management, access control and redundant file parsing mould
Block, described data analysis system 7 includes distributed scheduling unit 71, and described distributed scheduling unit 71 electrically inputs connection distribution
Formula computation model 72, described distributed computing platform 72 is electrically bi-directionally connected data conversion module 73, data aggregate unit respectively
74th, data associating unit 75 data excavates unit 76, and described data service system 9 includes distributed data bus 91, described
Distributed data bus 91 is electrically bi-directionally connected service management system 92, data access service 93 and service logic service respectively
94, described data access service 93 and service logic service 94 all electrically output connect data cache module 95, described data is visited
The service of asking 93 is internal to include database access unit, real-time data access unit and file system access unit, and described business is patrolled
Collect service 94 and include commending system, intelligently parsing module and statistical report form unit, described operation management unit 11 includes monitoring and passes
Sensor cell 111, data source 112 and instrument cluster 113, described monitor sensor unit 111, data source 112 and instrument cluster
113 all electrically export connection acquisition configuration unit 114, and described acquisition configuration unit 114 electrically exports connection via node 115,
Described via node 115 electrically exports connection signal amplification unit 116, and described signal amplification unit 116 electrically exports connection control
Terminal 117 processed.
Operation principle：Step system initialization in, whole system is initialized first, after the completion of execution system therefore
Barrier detecting step, if abnormal, returns previous step and is reinitialized, if normally, execute data analysis step,
Data analysis step is analyzed to big data, after the completion of carry out Data Integration, by various types of Data Integrations process
Afterwards, output is shown by data result, then complete this data management processes.
The full-service flow data of multiple types, multi-source is acquired obtain by data acquisition module 1, using data
Pretreatment unit 2 carries out hierarchical extraction, filtration etc. and processes, and data-storage system 3 can meet the base of coherence request
On plinth, the safe and reliable data to polymorphic type, multi-format, many characteristics stores, using data categorization module 4 data rope
Draw unit 5 and Classification And Index process is carried out to the data after storage, integrate scattered source data via Data Integration unit 12, look into
Ask access port 6 can external query facility to system information real-time query, by data analysis system 7, deposit from leaving data in
Value -capture information in a large amount of source datas in storage system 3 and user behavior data, sets up data mould using data modeling unit 8
Type, by data service system 9 via data result display unit 10 to analysis result conceptualization and systematization, and various dimensions and
Three-dimensional being shown of variation, using operation management unit 11 configuration and control authority management, monitoring alarm and fault pipe
Reason, in data-storage system 3, by integrated date managment system 31 integrated management integrated database 32, real-time data base 34
With distributed file system 35, issued the information in integrated database 32 by database synchronization release unit 33 in real time, counting
According in analysis system 7, distributed computing platform 72 is controlled by distributed scheduling unit 71, realizes data conversion.Data aggregate,
The data mining duty of data association data, comprehensively, analysis ability is strong, in data service system 9, using distributed for data analysiss
Data/address bus 91 controls service management system 92, data access service 93 and service logic unit 94 respectively, can be comprehensively right
The scattered source data of terminal carries out in-depth process, and implements to cache by data cache module 95, in operation management unit 11, leads to
Cross and the data message in monitoring sensor 111, data source 112 and instrument cluster 113 is adopted in real time via acquisition configuration unit 114
Collection, is amplified to signal source signal by via node 115 and signal amplification unit 116, and by the signal transmission after amplifying to control
In terminal 117 processed, this system disclosure satisfy that data management requirement necessary to the development of today's society information data expansion type.
Although an embodiment of the present invention has been shown and described, for the ordinary skill in the art, permissible
Understand and can carry out multiple changes, modification, replacement to these embodiments without departing from the principles and spirit of the present invention
And modification, the scope of the present invention be defined by the appended.
Claims (7)
1. a kind of big data management system, including data acquisition module (1) it is characterised in that：Described data acquisition module (1) electricity
Property output connect data pre-processing unit (2), electrically output connects data-storage system (3) to described data pre-processing unit (2),
Electrically output connection data categorization module (4) data indexing units (5) respectively of described data-storage system (3), described data
Sort module (4) data indexing units (5) all electrically output connection Data Integration unit (12), described Data Integration unit
(12) electrically export Connection inquiring access port (6) data analysis system (7) respectively, described data analysis system (7) is electrically
Output connects data modeling unit (8), and electrically output connects data service system (9), described number to described data modeling unit (8)
According to service system (9), electrically output connects data result display unit (10), and described data service system (9) is electrically bi-directionally connected
Operation management unit (11).
2. a kind of big data management system according to claim 1 it is characterised in that：Described data-storage system (3) bag
Include integrated date managment system (31), described integrated date managment system (31) is electrically bi-directionally connected integrated database respectively
(32), real-time data base (34) and distributed file system (35), electrically output connects data base to described integrated database (32)
Synchronous release unit (33).
3. a kind of big data management system according to claim 2 it is characterised in that：Described real-time data base (34) is internal
Including database management unit, memory database and hard disc data storehouse, described distributed file system (35) includes source data pipe
Reason, access control and redundant file parsing module.
4. a kind of big data management system according to claim 1 it is characterised in that：Described data analysis system (7) bag
Include distributed scheduling unit (71), electrically input connects distributed computing platform (72), institute to described distributed scheduling unit (71)
State distributed computing platform (72) and be electrically bi-directionally connected data conversion module (73), data aggregate unit (74), data pass respectively
Receipts or other documents in duplicate unit (75) data excavates unit (76).
5. a kind of big data management system according to claim 1 it is characterised in that：Described data service system (9) bag
Include distributed data bus (91), described distributed data bus (91) be electrically bi-directionally connected respectively service management system (92),
Data access service (93) and service logic service (94), described data access service (93) and service logic service (94) all electricity
Property output connect data cache module (95).
6. a kind of big data management system according to claim 5 it is characterised in that：In described data access service (93)
Portion includes database access unit, real-time data access unit and file system access unit, described service logic service (94)
Including commending system, intelligently parsing module and statistical report form unit.
7. a kind of big data management system according to claim 1 it is characterised in that：Described operation management unit (11) bag
Include monitor sensor unit (111), data source (112) and instrument cluster (113), described monitor sensor unit (111), data
Source (112) and instrument cluster (113) all electrically output connection acquisition configuration unit (114), described acquisition configuration unit (114) electricity
Property output connect via node (115), electrically output connects signal amplification unit (116), described letter to described via node (115)
Electrically output connects control terminal (117) for number amplifying unit (116).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610815529.8A CN106446085A (en) | 2016-09-09 | 2016-09-09 | Big data management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610815529.8A CN106446085A (en) | 2016-09-09 | 2016-09-09 | Big data management system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106446085A true CN106446085A (en) | 2017-02-22 |
Family
ID=58168440
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610815529.8A Pending CN106446085A (en) | 2016-09-09 | 2016-09-09 | Big data management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106446085A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107480287A (en) * | 2017-08-25 | 2017-12-15 | 成都优力德新能源有限公司 | Data integration device based on cloud computing |
| CN107766424A (en) * | 2017-09-13 | 2018-03-06 | 深圳市宇数科技有限公司 | A kind of Data Mining management method, system, electronic equipment and storage medium |
| CN107817948A (en) * | 2017-09-28 | 2018-03-20 | 新乡学院 | A kind of network teledata storage platform |
| CN108829704A (en) * | 2018-04-28 | 2018-11-16 | 安徽瑞来宝信息科技有限公司 | A kind of big data distributed libray Analysis Service technology |
| CN109033122A (en) * | 2018-05-30 | 2018-12-18 | 华存数据信息技术有限公司 | A kind of management method and device of big data software |
| CN109241758A (en) * | 2018-08-30 | 2019-01-18 | 安阳工学院 | A kind of big data analysis system using computer verification code technology |
| CN110278260A (en) * | 2019-06-17 | 2019-09-24 | 武汉灯塔之光科技有限公司 | A kind of forwarding recorded broadcast mthods, systems and devices of difference securities market data |
| CN110688645A (en) * | 2019-10-10 | 2020-01-14 | 安康学院 | Big data analysis system based on computer verification code technology |
| CN112765167A (en) * | 2021-01-06 | 2021-05-07 | 杭州哒卡乐智能科技有限公司 | Comprehensive data processing service system |
| CN114911862A (en) * | 2022-07-18 | 2022-08-16 | 国网江苏省电力有限公司营销服务中心 | System and method for transmitting big data of network operation link of network |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103646311A (en) * | 2013-12-06 | 2014-03-19 | 因为科技无锡有限公司 | Traffic data service system |
| CN104699849A (en) * | 2015-04-07 | 2015-06-10 | 同方知网数字出版技术股份有限公司 | Digital library resource unified search system |
| CN105488191A (en) * | 2015-12-03 | 2016-04-13 | 南京联成科技发展有限公司 | Data acquisition processing method and device based on big data information safety management operation and maintenance service platform |
| CN105809356A (en) * | 2016-03-14 | 2016-07-27 | 浪潮软件股份有限公司 | Information system resource management method based on application integrated cloud platform |
-
2016
- 2016-09-09 CN CN201610815529.8A patent/CN106446085A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103646311A (en) * | 2013-12-06 | 2014-03-19 | 因为科技无锡有限公司 | Traffic data service system |
| CN104699849A (en) * | 2015-04-07 | 2015-06-10 | 同方知网数字出版技术股份有限公司 | Digital library resource unified search system |
| CN105488191A (en) * | 2015-12-03 | 2016-04-13 | 南京联成科技发展有限公司 | Data acquisition processing method and device based on big data information safety management operation and maintenance service platform |
| CN105809356A (en) * | 2016-03-14 | 2016-07-27 | 浪潮软件股份有限公司 | Information system resource management method based on application integrated cloud platform |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107480287A (en) * | 2017-08-25 | 2017-12-15 | 成都优力德新能源有限公司 | Data integration device based on cloud computing |
| CN107766424A (en) * | 2017-09-13 | 2018-03-06 | 深圳市宇数科技有限公司 | A kind of Data Mining management method, system, electronic equipment and storage medium |
| CN107766424B (en) * | 2017-09-13 | 2020-09-15 | 深圳市宇数科技有限公司 | Data exploration management method and system, electronic equipment and storage medium |
| CN107817948A (en) * | 2017-09-28 | 2018-03-20 | 新乡学院 | A kind of network teledata storage platform |
| CN108829704A (en) * | 2018-04-28 | 2018-11-16 | 安徽瑞来宝信息科技有限公司 | A kind of big data distributed libray Analysis Service technology |
| CN109033122A (en) * | 2018-05-30 | 2018-12-18 | 华存数据信息技术有限公司 | A kind of management method and device of big data software |
| CN109241758A (en) * | 2018-08-30 | 2019-01-18 | 安阳工学院 | A kind of big data analysis system using computer verification code technology |
| CN110278260A (en) * | 2019-06-17 | 2019-09-24 | 武汉灯塔之光科技有限公司 | A kind of forwarding recorded broadcast mthods, systems and devices of difference securities market data |
| CN110688645A (en) * | 2019-10-10 | 2020-01-14 | 安康学院 | Big data analysis system based on computer verification code technology |
| CN112765167A (en) * | 2021-01-06 | 2021-05-07 | 杭州哒卡乐智能科技有限公司 | Comprehensive data processing service system |
| CN114911862A (en) * | 2022-07-18 | 2022-08-16 | 国网江苏省电力有限公司营销服务中心 | System and method for transmitting big data of network operation link of network |
| CN114911862B (en) * | 2022-07-18 | 2022-12-06 | 国网江苏省电力有限公司营销服务中心 | System and method for transmitting big data of network national network operation link |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106446085A (en) | Big data management system | |
| CN103500173B (en) | A kind of querying method of track traffic Monitoring Data | |
| CN106600103A (en) | Statistic data model building method facing programs, plans, and decisions | |
| CN112698953A (en) | Power grid intelligent operation and detection platform based on micro-service | |
| CN103559562A (en) | Power grid intelligent operation system and achieving method thereof | |
| CN106227846A (en) | Electric network information methods of exhibiting real-time and device | |
| CN103825755A (en) | Power secondary system modeling method and system | |
| CN112883001A (en) | Data processing method, device and medium based on marketing and distribution through data visualization platform | |
| CN109977125A (en) | A kind of big data safety analysis plateform system based on network security | |
| CN114238388A (en) | Heterogeneous data collection and retrieval system based on multiple protocols | |
| CN111159152B (en) | Secondary operation and data fusion method based on big data processing technology | |
| CN106407072A (en) | Monitoring system of big data platform | |
| CN113706333A (en) | Method and system for automatically generating topology island of power distribution network | |
| CN206850788U (en) | A kind of heterogeneous communication system integrates control platform | |
| CN105205185A (en) | Data interaction and data modeling method between monitoring system and management information system | |
| CN110133417A (en) | Grid line loss detection system and method | |
| CN102902741B (en) | A kind of method of communication network resource data cache and state cascading update | |
| CN107341198A (en) | A kind of electric power mass data storage and querying method based on subject example | |
| CN104408665B (en) | Based on the event-oriented noisy data processing system of SCD models | |
| CN114662925B (en) | Data fusion and mining method for power service center | |
| CN116094174A (en) | Knowledge graph-based power grid operation and maintenance monitoring method, system, equipment and medium | |
| CN114421617A (en) | Medium-voltage fault research and judgment method based on distribution network operation monitoring data | |
| CN112784129A (en) | Pump station equipment operation and maintenance data supervision platform | |
| CN113342874A (en) | Wind power big data analysis system and process based on cloud computing | |
| CN111177227A (en) | Power data self-service analysis system and decision application migration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170222 |