CN107943986A - A kind of big data analysis digging system - Google Patents
A kind of big data analysis digging system Download PDFInfo
- Publication number
- CN107943986A CN107943986A CN201711244293.8A CN201711244293A CN107943986A CN 107943986 A CN107943986 A CN 107943986A CN 201711244293 A CN201711244293 A CN 201711244293A CN 107943986 A CN107943986 A CN 107943986A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- data analysis
- mining
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
Abstract
The present invention discloses a kind of big data analysis digging system, including several data analysis modules, data analysis module manager, several data-mining modules, data-mining module manager and frame execution module, each data analysis module carries out respective simple function realization according to interface, while obtains data from raw data base and analyzed and be stored in respective module database；Multiple data analysis modules are managed collectively by data analysis module manager；Each data-mining module carries out respective simple function realization according to interface；Multiple data-mining modules are managed collectively by data-mining module manager；Frame execution module is parsed according to incoming structured document, and that extracts from data analysis module manager and data mining module management corresponding function realizes module, execution module function, thus obtain it is final needed for data.Such a system can save human resources, improve demand response efficiency, and excavating demand for different pieces of information provides facility.
Description
Technical field
The invention belongs to big data analysis field, more particularly to a kind of module based on structured documents such as XML and JSON
It polymerize big data analysis digging system.
Background technology
Under the historical background that information technology continues to develop, mass data all is being produced daily, big data analysis is in each row
Industry is widely used in field, or even gradually starts to penetrate into our daily life.The data analysis of high efficient and flexible
It is one of problem of big data system regions most study to excavate framework, its main effect is exactly the sea that will be stored in database
It is required big data report that amount data carry out effectively analyzing and neatly merging statistics as desired, corresponding to carry out with this
Meaningful judgement.
With the arrival of Internet era, network data is continuously increased, and analytic statistics is carried out to various network datas
Demand is also following, and the analysis in industry to data at present also rests on the progress special exploitation of particular analysis module according to demand
Stage on, flexible partition can not be carried out to data analysis mining demand in general form and integrated.Therefore, by various data
Analysis demand is split as multiple single realizing, and module and the system being polymerized with structured document are a feasible solution party
Case.
The content of the invention
The purpose of the present invention, is to provide a kind of big data analysis digging system, it can save human resources, improves demand
Response efficiency, excavates demand for different pieces of information and provides facility.
In order to achieve the above objectives, solution of the invention is：
A kind of big data analysis digging system, including several data analysis modules, data analysis module manager, several numbers
According to excavation module, data-mining module manager and frame execution module, wherein, data analysis module defines unified module and connects
Mouthful, each data analysis module carries out respective simple function realization according to interface, while each data analysis module is from initial data
Data are obtained in storehouse to be analyzed and be stored in respective module database；Multiple data analysis modules are by data analysis module pipe
Reason device is managed collectively；Data-mining module defines unified module interface, and each data-mining module carries out each according to interface
From simple function realize；Multiple data-mining modules are managed collectively by data-mining module manager；Frame performs mould
Block is responsible for being parsed according to incoming structured document, and according to file structure and parameter from data analysis module manager and
The module of realizing of corresponding function is extracted in data-mining module manager, structure forms aggregation module, according to pre-defined system
One analysis module interface excavates module interface execution module function with unified, so as to obtain final required data.
Above-mentioned each data-mining module is also built-in with initial data bank interface, for extracting original number from raw data base
According to.
Above-mentioned each data-mining module is also built-in with data analysis module manager, for being pre-processed from each analysis module
Data are extracted in database, and respective statistics is generated according to the data mining algorithm of realization.
Said structure document is XML or JSON document.
After using the above scheme, the present invention by the disassembling of data analysis requirements, the modular implementation of simple function and
Based on the final module polymerization of structured document, realize flexible data digging system, system framework code etc. can not changed
Under the premise of, by increasing analysis module and excavating module and the incoming file structure of adjustment, adapting to different data minings automatically needs
Ask, it is not necessary to excavate the corresponding function of demand overlapping development further according to different pieces of information, save human resources, improve demand response effect
Rate, excavates demand for different pieces of information and provides facility.The present invention is suitable for various data mining scenes.
Brief description of the drawings
Fig. 1 is the integrated stand composition of the present invention.
Embodiment
Below with reference to attached drawing, technical scheme is described in detail.
As shown in Figure 1, the present invention provides a kind of big data analysis digging system, including data analysis module, data mining
Module and frame execution module, wherein, data analysis module defines unified module interface, respectively realizes that module is carried out according to interface
Respective simple function is realized, while each module obtains data from raw data base and analyzed and be stored in respective number of modules
According in storehouse, multiple data analysis modules are managed collectively by data analysis module manager.Data-mining module definition is unified
Module interface, respectively realize that module carries out respective simple function realization according to interface, while built-in initial data bank interface,
For extracting initial data, and built-in data analysis module manager from raw data base, for pre- from each analysis module
Data are extracted in processing database, and respective statistics, multiple data minings are generated according to the data mining algorithm of realization
Module is managed collectively by data-mining module manager.Frame execution module (namely aggregation module actuator in Fig. 1)
It is mainly responsible for and is parsed according to structured documents such as incoming XML or JSON, and is divided according to file structure and parameter from data
That corresponding function is extracted in analysis module management and data mining module management realizes that module is built, and will finally build
Aggregation module be passed in aggregation module actuator, aggregation module actuator according to pre-defined united analysis module interface with
It is unified to excavate module interface execution module function, so as to obtain final required data.
To sum up, the present invention obtains corresponding function module by the module title identified in parameter first；Secondly, according to mark
Corresponding module parameter carry out data mining；Finally, the data of excavation are integrated according to file structure, so as to reach generation
Meet the purpose of the data of demand.
Above example is merely illustrative of the invention's technical idea, it is impossible to protection scope of the present invention is limited with this, it is every
According to technological thought proposed by the present invention, any change done on the basis of technical solution, each falls within the scope of the present invention
Within.
Claims (4)
- A kind of 1. big data analysis digging system, it is characterised in that：Including several data analysis modules, data analysis module management Device, several data-mining modules, data-mining module manager and frame execution module, wherein, data analysis module definition system One module interface, each data analysis module carry out respective simple function realization, while each data analysis module according to interface Data are obtained from raw data base to be analyzed and be stored in respective module database；Multiple data analysis modules are by data Analysis module manager is managed collectively；Data-mining module defines unified module interface, each data-mining module according to Interface carries out respective simple function realization；Multiple data-mining modules are managed collectively by data-mining module manager； Frame execution module is responsible for being parsed according to incoming structured document, and according to file structure and parameter from data analysis mould The module of realizing of corresponding function is extracted in block manager and data mining module management, structure forms aggregation module, according to pre- The united analysis module interface first defined excavates module interface execution module function with unified, so as to obtain final required data.
- A kind of 2. big data analysis digging system as claimed in claim 1, it is characterised in that：Each data-mining module is also Initial data bank interface is built-in with, for extracting initial data from raw data base.
- A kind of 3. big data analysis digging system as claimed in claim 1, it is characterised in that：Each data-mining module is also Data analysis module manager is built-in with, for extracting data from each analysis module preprocessed data storehouse, and according to realization Data mining algorithm generate respective statistics.
- A kind of 4. big data analysis digging system as claimed in claim 1, it is characterised in that：The structured document is XML Or JSON document.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711244293.8A CN107943986B (en) | 2017-11-30 | 2017-11-30 | Big data analysis mining system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711244293.8A CN107943986B (en) | 2017-11-30 | 2017-11-30 | Big data analysis mining system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107943986A true CN107943986A (en) | 2018-04-20 |
| CN107943986B CN107943986B (en) | 2022-05-17 |
Family
ID=61947156
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711244293.8A Active CN107943986B (en) | 2017-11-30 | 2017-11-30 | Big data analysis mining system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107943986B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5692107A (en) * | 1994-03-15 | 1997-11-25 | Lockheed Missiles & Space Company, Inc. | Method for generating predictive models in a computer system |
| US20020169735A1 (en) * | 2001-03-07 | 2002-11-14 | David Kil | Automatic mapping from data to preprocessing algorithms |
| CN1399228A (en) * | 2002-08-29 | 2003-02-26 | 北京北大方正技术研究院有限公司 | Text excavating method of semi-structural document set |
| US6865573B1 (en) * | 2001-07-27 | 2005-03-08 | Oracle International Corporation | Data mining application programming interface |
| CN103577605A (en) * | 2013-11-20 | 2014-02-12 | 贵州电网公司电力调度控制中心 | Data warehouse based on data fusion and data mining and application method of data warehouse |
| CN107025288A (en) * | 2017-04-14 | 2017-08-08 | 四川九鼎瑞信软件开发有限公司 | Distributed data digging method and system |
| CN107038167A (en) * | 2016-02-03 | 2017-08-11 | 普华诚信信息技术有限公司 | Big data excavating analysis system and its analysis method based on model evaluation |
| CN107357873A (en) * | 2017-07-04 | 2017-11-17 | 深圳齐心集团股份有限公司 | A kind of big data storage management system |
-
2017
- 2017-11-30 CN CN201711244293.8A patent/CN107943986B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5692107A (en) * | 1994-03-15 | 1997-11-25 | Lockheed Missiles & Space Company, Inc. | Method for generating predictive models in a computer system |
| US20020169735A1 (en) * | 2001-03-07 | 2002-11-14 | David Kil | Automatic mapping from data to preprocessing algorithms |
| US6865573B1 (en) * | 2001-07-27 | 2005-03-08 | Oracle International Corporation | Data mining application programming interface |
| CN1399228A (en) * | 2002-08-29 | 2003-02-26 | 北京北大方正技术研究院有限公司 | Text excavating method of semi-structural document set |
| CN103577605A (en) * | 2013-11-20 | 2014-02-12 | 贵州电网公司电力调度控制中心 | Data warehouse based on data fusion and data mining and application method of data warehouse |
| CN107038167A (en) * | 2016-02-03 | 2017-08-11 | 普华诚信信息技术有限公司 | Big data excavating analysis system and its analysis method based on model evaluation |
| CN107025288A (en) * | 2017-04-14 | 2017-08-08 | 四川九鼎瑞信软件开发有限公司 | Distributed data digging method and system |
| CN107357873A (en) * | 2017-07-04 | 2017-11-17 | 深圳齐心集团股份有限公司 | A kind of big data storage management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107943986B (en) | 2022-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102222092B (en) | Massive high-dimension data clustering method for MapReduce platform | |
| CN106709012A (en) | Method and device for analyzing big data | |
| CN102222105B (en) | Method for generating real-time statistical report | |
| CN104915378B (en) | A kind of statistics task quick-speed generation system and method suitable for big data | |
| CN107577771A (en) | A kind of big data digging system | |
| CN110750650A (en) | Construction method and device of enterprise knowledge graph | |
| CN104598565B (en) | A kind of K mean value large-scale data clustering methods based on stochastic gradient descent algorithm | |
| CN103440566A (en) | Method and device for generating order picking collection lists and method for optimizing order picking route | |
| US11520825B2 (en) | Method and system for converting one type of data schema to another type of data schema | |
| CN106959948A (en) | The system and its preprocess method pre-processed for distributed nature to big data | |
| CN103853826B (en) | A kind of distributed performance data processing method | |
| CN105574032A (en) | Rule matching operation method and device | |
| CN107967347A (en) | Batch data processing method, server, system and storage medium | |
| CN106789347A (en) | A kind of method that alarm association and network fault diagnosis are realized based on alarm data | |
| CN105760511B (en) | A kind of big data adaptive topology processing method based on storm | |
| CN105138650A (en) | Hadoop data cleaning method and system based on outlier mining | |
| CN103425692B (en) | Data export method and device | |
| CN108829884A (en) | data mapping method and device | |
| CN104143122A (en) | Intelligent service approval scheme | |
| CN1831855A (en) | Real-time analysing and control system and method for large underground hole group construction at network environment | |
| CN109829660A (en) | Data processing system and its design method based on electric power enterprise grade data model | |
| CN106657099A (en) | Spark data analysis service release system | |
| CN107590225A (en) | A kind of Visualized management system based on distributed data digging algorithm | |
| CN105391777A (en) | Algorithm escrow PaaS platform for decoupling logic code and performance code | |
| CN104166701A (en) | Machine learning method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |