CN109189764A - A kind of colleges and universities' data warehouse layered design method based on Hive - Google Patents
A kind of colleges and universities' data warehouse layered design method based on Hive Download PDFInfo
- Publication number
- CN109189764A CN109189764A CN201811098136.5A CN201811098136A CN109189764A CN 109189764 A CN109189764 A CN 109189764A CN 201811098136 A CN201811098136 A CN 201811098136A CN 109189764 A CN109189764 A CN 109189764A
- Authority
- CN
- China
- Prior art keywords
- data
- theme
- student
- analysis
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention relates to a kind of colleges and universities' data warehouse layered design method based on Hive, comprising: obtain data, carry out data pick-up using ETL tool, the structuring that will acquire, unstructured data are synchronized on Hive platform；Data warehouse is constructed using Hive, data warehouse is divided into ODS data storage layer, DWD data detail layer, DW data summarization layer, DWA data application layer；Data warehouse modeling determines analysis theme, using dimensionality analysis method, designs dimension table using minimum particle size, designs true table；True table is designed, true table-case of non-partitioned tables and subregion fact table are divided into.Data warehouse hierarchical design proposed by the present invention handles more flexible compared to other three layer analysis of big data warehouse design, scalability is strong, later period can increase corresponding analysis theme according to business demand, and Hive big data platform advantage and data warehouse Star Model design method are efficiently combined.
Description
Technical field
The invention belongs to database technical fields, and in particular to a kind of data warehouse hierarchical design side of colleges and universities based on Hive
Method.
Background technique
Desired continuous promotion is reached its maturity and managed with university information system construction, data warehouse can be introduced
Technology carries out structural rearrangement to university information system data, the characteristics of for colleges and universities and growth requirement, by being more advantageous to decision point
The angle of analysis goes to design, and the analysis such as data mining is carried out on data warehouse, and the data resource for making these valuable is realized real
Information value, improve to the utilization rate of management information data, and then it is horizontal to promote university managementt.
Hive is a Tool for Data Warehouse based on Hadoop, the data file of structuring can be mapped as a number
According to library table, and simple sql query function is provided, sql sentence can be converted to MapReduce task and run.
Hadoop is a distributed system infrastructure developed by apache foundation.Data warehouse (DW, Data
Warehouse) it is a subject-oriented, integrated, changing over time, metastable data acquisition system, is used for stay pipe
Manage decision.By constructing data warehouse, functional department to the data of existing system can effectively integrate and recombinated, build
Facade reports demand to the system of on-line analytical processing to meet school to the accurate grasp of data, statistical analysis, and be data
It excavates and decision support provides basis.
Traditional data warehouse is broadly divided into ODS data storage layer (substantially preservation full dose data)-DW data warehouse layer-
DM (Data Market) data set city level, traditional triple layer designs framework cannot achieve increment+full dose data method of synchronization, be
Data complex logic is all placed on DW layers, flexibility is poor.
Summary of the invention
The object of the invention is that a kind of colleges and universities' data based on Hive proposed for the defects of background technique
Warehouse layered design method.
Data warehouse is a subject-oriented, integrated, changing over time, metastable data acquisition system, is used for
Support administrative decision.By constructing data warehouse (Data Warehouse), functional department can carry out the data of existing system
It is effective to integrate and recombinated, the system towards on-line analytical processing is established, to meet school to the accurate grasp of data, system
Meter analysis reports demand, and provides basis for data mining and decision support.The definition of one complete data warehouse is:
Data warehouse (DWS (Data Warehouse System)=extraction/conversion/load (ETL)+data warehouse (DW)+connection
Machine analysis handles (OLAP)+data mining (DM)+decision support (DS).
As big data platform hadoop is continued to develop, the Hive data warehouse on hadoop platform provides a system
The tool of column can be used to carry out data to extract conversion load (ETL), wherein ETL is that one kind can store, inquires and analyze
It is stored in the mechanism of the large-scale data in Hadoop.Colleges and universities' data warehouse hierarchical design based on Hive, can meet well
The quick increase of College Informatization fast development and business datum amount, and there is scalability well, both meet present colleges
Service management demand also provides extension function for follow-up business regulatory requirement, and therefore, Hive is to be most suitable for data warehouse applications journey
Sequence, it can safeguard mass data, and can excavate to data, then form opinion and report etc..
The present invention through the following technical solutions to achieve the above objectives:
A kind of colleges and universities' data warehouse layered design method based on Hive, comprising the following steps:
Step 1, data are obtained, from work system, education administration system, card system, subsidize system, network log-in management system
System, campus wireless system, personnel system, attendance checking system, access control system, Dormitory management system, financial system, obtain structuring with
Non-structured data；
Step 2, data pick-up is carried out using ETL (Extract-Transform-Load is loaded according to conversion is extracted) tool,
The structuring that will acquire, unstructured data are synchronized on Hive platform；
Step 3, using Hive construct data warehouse, by data warehouse be divided into ODS data storage layer, DWD data detail layer,
DW data summarization layer, DWA (Data Warehouse Application) data application layer；
Wherein ODS (Operational Data Store Operational data store library) data storage layer is data buffer storage layer,
For storing the initial data obtained, retains a regular length time, any processing is not done to data；
Wherein DWD (Data Warehouse Detail) data detail layer is used to carry out the data of ODS data storage layer
Cleaning, transcoding, increment turn full dose, store after carrying out unified standard with field name to table name word；The layer data granularity and ODS mono-
It causes, can be used as the basic data of access, analysis, excavation.DWD layers of transcoding need to correspond with source system, and dimension is forbidden to restrain；
Wherein DW data summarization layer is used for subject-oriented group organization data, according to requirements of service constructs multidimensional model data, carries out
The fractionation of Data Integration, related service in related subject domain summarizes；For data granularity, the data of this layer are to summarize grade
Data and vertical wide table data still cover all business datums for the range of data；This layer further includes dimension table,
Started with DIM, dimension table includes common dimension and business dimension, wherein common dimension time dimension, region dimension etc., special such as school
The dimensions such as industry, class, department, student, trade company；
Wherein DWA data application layer is used to need to construct multidimensional model data according to service application, and the data obtained is directly used
Show in analysis, this layer also takes on the construction of thematic class data model；This layer also takes on the construction of thematic class data model simultaneously.
Step 4, data warehouse modeling determines analysis theme, using dimensionality analysis method, designs dimension table using minimum particle size,
Design true table；
The modeling method of more popular data warehouse is more at present, and there are commonly the normal form modelings that Inmon is advocated
The dimensionality analysis method advocated with Kimball.Dimensionality analysis method has done a large amount of pretreatment for each dimension, passes through these pre- places
Reason can greatly promote the processing capacity of data warehouse, for normal form modeling, occupy in performance apparent
Advantage；Dimensionality analysis is very intuitive simultaneously, tightened around business model, can intuitively reflect the business in business model
Problem.Dimensionality analysis can be completed by needing not move through special abstract processing.Therefore the number of colleges and universities' data statistics service platform
The mode of dimensionality analysis is taken to construct according to warehouse.Dimensionality analysis method constructs data warehouse by the way of true table-dimension table, number
Actual data are stored according to fairground, true table, dimension table stores the attribute of object in true table, the incidence relation of true table and dimension table
Have 3 kinds of " Star Model ", " snowflake model " and " mixed model ", the most commonly used is " Star Models ", so using Star Model come
Modeling.
True table is designed, true table-case of non-partitioned tables and subregion fact table are divided into.
The present invention further improvement lies in that, step 2 specifically includes the following steps:
Step 2.1, ETL tool selection open source Kettle or Sqoop；
Step 2.2, the selection of mode is extracted, few for data volume, change is measured big data source and is extracted using full dose is synchronous,
It is big to data volume, it changes small data source and increment synchronization is taken to extract；
Based on source table date and time stamp or renewal time as subregion field, increment extraction is carried out according to time subregion,
Full dose is used to extract if without time type field；Increment+full dose is synchronous to be extracted, and Hive data warehouse partition table is made full use of
Advantage；
Step 2.3, standardized to data, verified, cleaned；
Step 2.4, the log that record ETL is extracted；
Step 2.5, when ETL tool issues abnormal notice, maintenance people is sent mail to after capturing using ETL built-in tool
Member.
The present invention further improvement lies in that, step 4 include it is following step by step:
Step 4.1, determine analysis theme, the analysis theme include a common dimension theme, further include student's theme,
School work theme, consumption theme, subsidizes theme, gate inhibition's theme, attendance theme, wireless theme, online theme at dormitory theme；
Common dimension theme includes time dimension, region dimension, national standard and school mark dimension；Different application scenarios can be with
Specific analysis dimension is converted to using view, national standard is mainly used to solve consistent during data integration with school mark
Property problem；
Step 4.2, dimension table is designed using minimum particle size, using entity as an object when choosing dimension, right with this
As the extraction of relevant important attribute, as independent dimension；It determines analysis granularity, is generally exactly the detailed journey for analyzing object
Degree.In order to meet the scalability of analysis and the diversity of demand, carrying out design data model always with minimum particle size can reach most
Good analysis effect, such as: recording the detail situation of each student, consumption details data are accurate to the specific consumption newest granularity of Hour Minute Second
Data.
Step 4.3, true table is designed, small, the big data of data volume are changed in storage in subregion fact table；True table-is overstepping one's bounds
Area's table stores student's basic information.
The change of subregion fact Biao Zhong colleges and universities' major part system data is big compared with small but data volume, such as all-purpose card consumption and online
User behaviors log etc. includes date day_id, month month_id and time year_id, this part is filled according to time partitioned storage
Ground is divided to realize the increment extraction to data using the partition table advantage of Hive Data Warehouse Platform；
True table-case of non-partitioned tables: be directed to the basic information such as student's essential information, using full dose extraction by the way of into
Row, to realize full dose+increment mixed synchronization decimation pattern for colleges and universities' business scenario well.
The present invention further improvement lies in that, student's theme core content is the basic condition of student, make a concrete analysis of student institute
In source of students, gender, nationality, political affiliation, health status, class, profession, department, academic year, length of schooling, educational background；
Wherein school work theme core content is student performance learning information, makes a concrete analysis of student's curriculum information, achievement,
Divide, point, learn duration and library loan information；
Wherein dormitory theme core content is student's lodging information, concrete analysis include student where dormitory building, room number,
Bed and accommodation electricity usage situation；
Wherein consumption theme core content is student's all-purpose card consumption, makes a concrete analysis of student in dining room, supermarket, books
Shop, fruit shop, boiling water room, computer room, hospital, bathroom consumption type overall condition；
Wherein subsidize theme core content be student obtain prize supplementary information situation, concrete analysis include scholarship, scholarship,
Loans for supporting students are taken a part-time job while studying at school, the subsidy situation of tuition waiver type；
Wherein gate inhibition's theme core content is the discrepancy passage situation of student, and concrete analysis module includes dormitory disengaging gate inhibition
Data, library pass in and out gate inhibition's data；
Wherein attendance theme core content is that student attends class situation, and concrete analysis includes whether to attend class on time, the rate of attendance, late
To, leave early, situation of cutting classes；
Wherein wireless theme core content is students ' behavior track, and time and the position of access terminals are connected by student,
Action trail in analysis student one day, such as dormitory-dining room-teaching building-library-dining room-boiling water room-bathroom are similar
Action trail；
Wherein online theme core is network playing by students behavior situation, and concrete analysis includes online duration, network access style, online
Preference, search key.
This programme formulates data warehouse standard, is based on data warehouse metadata management, formulates according to colleges and universities' business corresponding
Data standard and standard, and be described in the design of data warehouse layered sheet, it is external to data application layer from authority data source inlet
Interface egress realizes the normalization, consistency and validity of data.
The beneficial effects of the invention are that traditional data warehouse is broadly divided into ODS data compared to traditional Based Data Warehouse System
Accumulation layer-DW data warehouse layer-DM data set city level, it is synchronous that traditional triple layer designs framework cannot achieve increment+full dose data
Mode is that data complex logic is all placed on DW layers, and flexibility is poor.The present invention uses four layers of design scheme, compared with other big numbers
More flexible according to the processing of three layer analysis of warehouse design, scalability is strong, and the later period can increase corresponding analysis theme according to business demand,
Hive big data platform advantage and data warehouse Star Model design method are efficiently combined.
Detailed description of the invention
Fig. 1 is overall structure diagram of the invention.
Specific embodiment
The application is described in further detail with reference to the accompanying drawing, it is necessary to it is indicated herein to be, implement in detail below
Mode is served only for that the application is further detailed, and should not be understood as the limitation to the application protection scope, the field
Technical staff can make some nonessential modifications and adaptations to the application according to above-mentioned application content.
Embodiment 1
It is a kind of colleges and universities' data warehouse frame such as Fig. 1, entire frame is divided into four layers, is data source, data storage respectively
Layer, data analysis layer and data application layer.
Wherein data source includes the data from each system of school, format include structuring table and non-structured day
Will data；
ETL tool such as Sqoop tool or open source kettle by data cleansing in data source, are converted, are loaded into Hadoop points
On cloth platform, Hdfs (distributed file system) distributed storage, Hive distributed treatment are used；
The data of data storage layer are established into data warehouse i.e. data analysis layer by Hive tool, wherein data warehouse point
For ODS data storage layer, DWD data detail layer, DW data summarization layer, DWA data application layer；
Wherein ODS data storage layer be data buffer storage layer, for store acquisition initial data, retain a regular length
Time does not do any processing to data；
Wherein DWD (detail) data detail layer for the data of ODS data storage layer are cleaned, transcoding, increment
Turn full dose, is stored after carrying out unified standard with field name to table name word；
Wherein DW data summarization layer is used for subject-oriented group organization data, according to requirements of service constructs multidimensional model data, carries out
The fractionation of Data Integration, related service in related subject domain summarizes；Including DW subject heading list and DIM dimension table；
Wherein DWA data application layer is used to need to construct multidimensional model data according to service application, and the data obtained is directly used
Show in analysis, this layer also takes on the construction of thematic class data model；
Wherein DWD layers running specifically includes the following steps:
Step S2.1, ETL tool selection open source Kettle or Sqoop；
Step S2.2 extracts the selection of mode, few for data volume, and change is measured big data source and taken out using full dose is synchronous
It takes, it is big to data volume, it changes small data source and increment synchronization is taken to extract；
Based on source table date and time stamp or renewal time as subregion field, increment extraction is carried out according to time subregion,
Full dose is used to extract if without time type field；Increment+full dose is synchronous to be extracted, and Hive data warehouse partition table is made full use of
Advantage；
Step S2.3 standardizes to data, is verified, is cleaned；
Step S2.4, the log that record ETL is extracted；
When step S2.5, ETL tool issues abnormal notice, maintenance people is sent mail to after capturing using ETL built-in tool
Member.
Complete data analysis layer design after, data warehouse is modeled using Hive tool, including it is following step by step:
Step S4.1 determines that analysis theme, the analysis theme include a common dimension theme, further includes student master
Topic, consumption theme, subsidizes theme, gate inhibition's theme, attendance theme, wireless theme, online theme at school work theme, dormitory theme；
Common dimension theme includes time dimension, region dimension, national standard and school mark dimension；
Step S4.2 designs dimension table using minimum particle size, using entity as an object when choosing dimension, right with this
As the extraction of relevant important attribute, as independent dimension；
Step S4.3 designs true table, and small, the big data of data volume are changed in storage in subregion fact table；True table-is overstepping one's bounds
Area's table stores student's basic information.
Data after modeling can submit to On Line Analysis Process, data mining DM, decision branch by ETL tool
DS use is held, according to theme difference, obtains reasonable conclusion.
The embodiments described above only express several embodiments of the present invention, and the description thereof is more specific and detailed, but simultaneously
Limitations on the scope of the patent of the present invention therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to guarantor of the invention
Protect range.
Claims (4)
1. a kind of colleges and universities' data warehouse layered design method based on Hive, which comprises the following steps:
Step 1, obtain data, from learn work system, education administration system, card system, subsidize system, network log-in management system,
Campus wireless system, personnel system, attendance checking system, access control system, Dormitory management system, financial system, obtain structuring with it is non-
The data of structuring；
Step 2, data pick-up is carried out using ETL tool, the structuring that will acquire, unstructured data are synchronized to Hive platform
On；
Step 3, data warehouse is constructed using Hive, data warehouse is divided into ODS data storage layer, DWD data detail layer, DW number
According to summarizing layer, DWA data application layer；
Wherein ODS data storage layer is data buffer storage layer, for storing the initial data obtained, when retaining a regular length
Between, any processing is not done to data；
Wherein DWD data detail layer is used to clean the data of ODS data storage layer, transcoding, increment turn full dose, to table name
Word stores after carrying out unified standard with field name；
Wherein DW data summarization layer is used for subject-oriented group organization data, according to requirements of service constructs multidimensional model data, carries out related
The fractionation of Data Integration, related service in subject area summarizes；
Wherein DWA data application layer is used to need to construct multidimensional model data according to service application, and the data obtained is directly used in point
Analysis shows, this layer also takes on the construction of thematic class data model；
Step 4, data warehouse modeling determines analysis theme, using dimensionality analysis method, designs dimension table, design using minimum particle size
True table；
True table is designed, true table-case of non-partitioned tables and subregion fact table are divided into.
2. a kind of colleges and universities' data warehouse layered design method based on Hive according to claim 1, which is characterized in that step
Rapid 2 specifically includes the following steps:
Step 2.1, ETL tool selection open source Kettle or Sqoop；
Step 2.2, the selection of mode is extracted, few for data volume, change is measured big data source and extracted using full dose is synchronous, logarithm
It is big according to amount, it changes small data source and increment synchronization is taken to extract；
Based on source table date and time stamp or renewal time as subregion field, increment extraction is carried out according to time subregion, if not having
Having time type field then uses full dose to extract；
Step 2.3, standardized to data, verified, cleaned；
Step 2.4, the log that record ETL is extracted；
Step 2.5, when ETL tool issues abnormal notice, maintenance personnel is sent mail to after capturing using ETL built-in tool.
3. a kind of colleges and universities' data warehouse layered design method based on Hive according to claim 1, which is characterized in that step
Rapid 4 include it is following step by step:
Step 4.1, it determines that analysis theme, the analysis theme include a common dimension theme, further includes student's theme, school work
Theme, consumption theme, subsidizes theme, gate inhibition's theme, attendance theme, wireless theme, online theme at dormitory theme；
Common dimension theme includes time dimension, region dimension, national standard and school mark dimension；
Step 4.2, design dimension table using minimum particle size, using entity as an object when choosing dimension, with the object phase
The important attribute of pass is extracted, as independent dimension；
Step 4.3, true table is designed, small, the big data of data volume are changed in storage in subregion fact table；True table-case of non-partitioned tables
Store student's basic information.
4. a kind of colleges and universities' data warehouse layered design method based on Hive according to claim 3, which is characterized in that learn
Raw theme core content is the basic condition of student, source of students where concrete analysis student, gender, nationality, political affiliation, health
Situation, class, profession, department, academic year, length of schooling, educational background；
Wherein school work theme core content is student performance learning information, makes a concrete analysis of student's curriculum information, achievement, credit, achievement
Point, study duration and library loan information；
Wherein dormitory theme core content is student's lodging information, and concrete analysis includes dormitory building, room number, bed where student
With accommodation electricity usage situation；
Wherein consumption theme core content be student's all-purpose card consumption, concrete analysis student dining room, supermarket, library,
Fruit shop, boiling water room, computer room, hospital, bathroom consumption type overall condition；
Wherein subsidizing theme core content is that student obtains prize supplementary information situation, and concrete analysis includes scholarship, scholarship, gives financial aid to students
It provides a loan, take a part-time job while studying at school, the subsidy situation of tuition waiver type；
Wherein gate inhibition's theme core content is the discrepancy passage situation of student, and concrete analysis module includes dormitory disengaging gate inhibition's number
Gate inhibition's data are passed in and out according to, library；
Wherein attendance theme core content is that student attends class situation, and concrete analysis includes whether to attend class on time, the rate of attendance, it is late,
It leaves early, situation of cutting classes；
Wherein wireless theme core content is students ' behavior track, and time and the position of access terminals, analysis are connected by student
Action trail in student one day, such as dormitory-dining room-teaching building-library-dining room-similar behavior in boiling water room-bathroom
Track；
Wherein online theme core is network playing by students behavior situation, and concrete analysis is inclined including online duration, network access style, online
Good, search key.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811098136.5A CN109189764A (en) | 2018-09-20 | 2018-09-20 | A kind of colleges and universities' data warehouse layered design method based on Hive |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811098136.5A CN109189764A (en) | 2018-09-20 | 2018-09-20 | A kind of colleges and universities' data warehouse layered design method based on Hive |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109189764A true CN109189764A (en) | 2019-01-11 |
Family
ID=64908571
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811098136.5A Pending CN109189764A (en) | 2018-09-20 | 2018-09-20 | A kind of colleges and universities' data warehouse layered design method based on Hive |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109189764A (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110119391A (en) * | 2019-05-14 | 2019-08-13 | 重庆八戒传媒有限公司 | A kind of data warehouse creation method and data warehouse based on service data |
| CN110222123A (en) * | 2019-04-24 | 2019-09-10 | 深圳点猫科技有限公司 | The method and electronic equipment that Hive dynamic partition for teaching platform loads |
| CN110263052A (en) * | 2019-06-25 | 2019-09-20 | 苏宁消费金融有限公司 | One kind automating simultaneous techniques innovative approach based on big data Hadoop platform ODS |
| CN110334088A (en) * | 2019-07-11 | 2019-10-15 | 江苏曲速教育科技有限公司 | Educational data management system |
| CN110850824A (en) * | 2019-11-12 | 2020-02-28 | 北京矿冶科技集团有限公司 | Implementation method for acquiring data of distributed control system to Hadoop platform |
| CN111008234A (en) * | 2019-11-27 | 2020-04-14 | 杭州安恒信息技术股份有限公司 | Warehouse processing method based on network safety data management |
| CN111143465A (en) * | 2019-12-11 | 2020-05-12 | 深圳市中电数通智慧安全科技股份有限公司 | Method and device for realizing data center station and electronic equipment |
| CN111259068A (en) * | 2020-04-28 | 2020-06-09 | 成都四方伟业软件股份有限公司 | Data development method and system based on data warehouse |
| CN111460045A (en) * | 2020-03-02 | 2020-07-28 | 心医国际数字医疗系统(大连)有限公司 | Modeling method, model, computer device and storage medium for data warehouse construction |
| CN111461621A (en) * | 2020-04-13 | 2020-07-28 | 郑州工程技术学院 | Distributed school financial management system, method, equipment and storage medium |
| CN111475528A (en) * | 2020-03-23 | 2020-07-31 | 深圳市酷开网络科技有限公司 | OTT-based data warehouse construction method, equipment and storage medium |
| CN111639121A (en) * | 2020-04-07 | 2020-09-08 | 国网新疆电力有限公司 | Big data platform and method for constructing customer portrait |
| CN111680108A (en) * | 2019-03-11 | 2020-09-18 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and data acquisition method and device |
| CN111694810A (en) * | 2019-03-12 | 2020-09-22 | 阿里巴巴集团控股有限公司 | Data warehouse creation method and device, electronic equipment and readable storage medium |
| CN112084182A (en) * | 2020-09-10 | 2020-12-15 | 重庆富民银行股份有限公司 | Data modeling method for data mart and data warehouse |
| CN112148807A (en) * | 2020-09-28 | 2020-12-29 | 中国电波传播研究所（中国电子科技集团公司第二十二研究所） | Electromagnetic environment field data warehouse construction method |
| CN112231301A (en) * | 2020-10-21 | 2021-01-15 | 黄河水利委员会黄河水利科学研究院 | Yellow river water sand change data warehouse |
| CN112380218A (en) * | 2020-11-18 | 2021-02-19 | 浪潮天元通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112687097A (en) * | 2020-11-16 | 2021-04-20 | 招商新智科技有限公司 | Highway highway section level data center platform system |
| CN112860659A (en) * | 2021-01-18 | 2021-05-28 | 北京奇艺世纪科技有限公司 | Data warehouse construction method, device, equipment and storage medium |
| CN112966024A (en) * | 2021-03-12 | 2021-06-15 | 江苏苏伦大数据科技研究院有限公司 | Financial wind control data analysis system based on big data |
| CN112988919A (en) * | 2021-04-30 | 2021-06-18 | 广东电网有限责任公司 | Power grid data market construction method and system, terminal device and storage medium |
| CN113486096A (en) * | 2021-06-21 | 2021-10-08 | 上海百秋电子商务有限公司 | Multi-library timing execution report data preprocessing and query method and system |
| CN113515362A (en) * | 2021-07-12 | 2021-10-19 | 广州云从洪荒智能科技有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN114385121A (en) * | 2022-01-13 | 2022-04-22 | 浙江工企信息技术股份有限公司 | Software design modeling method and system based on business layering |
| CN114595294A (en) * | 2022-03-11 | 2022-06-07 | 北京梦诚科技有限公司 | Data warehouse modeling and extracting method and system |
| CN114880405A (en) * | 2022-03-31 | 2022-08-09 | 华能信息技术有限公司 | Data lake-based data processing method and system |
| CN115618842A (en) * | 2022-12-15 | 2023-01-17 | 浙江蓝鸽科技有限公司 | Integrated intelligent campus data center system |
| CN116737846A (en) * | 2023-05-31 | 2023-09-12 | 深圳华夏凯词财富管理有限公司 | Asset management data safety protection warehouse system based on Hive |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101197876A (en) * | 2006-12-06 | 2008-06-11 | 中兴通讯股份有限公司 | Method and system for multi-dimensional analysis of message service data |
| US8417715B1 (en) * | 2007-12-19 | 2013-04-09 | Tilmann Bruckhaus | Platform independent plug-in methods and systems for data mining and analytics |
| CN104915456A (en) * | 2015-07-03 | 2015-09-16 | 宁夏隆基宁光仪表有限公司 | Mass power utilization data mining method on the basis of data analysis system |
| CN105184642A (en) * | 2015-09-02 | 2015-12-23 | 浪潮软件集团有限公司 | Comprehensive tax administration platform |
| WO2017040209A1 (en) * | 2015-08-31 | 2017-03-09 | BloomReach, Inc. | Data preparation for data mining |
| CN108280084A (en) * | 2017-01-06 | 2018-07-13 | 上海前隆信息科技有限公司 | A kind of construction method of data warehouse, system and server |
-
2018
- 2018-09-20 CN CN201811098136.5A patent/CN109189764A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101197876A (en) * | 2006-12-06 | 2008-06-11 | 中兴通讯股份有限公司 | Method and system for multi-dimensional analysis of message service data |
| US8417715B1 (en) * | 2007-12-19 | 2013-04-09 | Tilmann Bruckhaus | Platform independent plug-in methods and systems for data mining and analytics |
| CN104915456A (en) * | 2015-07-03 | 2015-09-16 | 宁夏隆基宁光仪表有限公司 | Mass power utilization data mining method on the basis of data analysis system |
| WO2017040209A1 (en) * | 2015-08-31 | 2017-03-09 | BloomReach, Inc. | Data preparation for data mining |
| CN105184642A (en) * | 2015-09-02 | 2015-12-23 | 浪潮软件集团有限公司 | Comprehensive tax administration platform |
| CN108280084A (en) * | 2017-01-06 | 2018-07-13 | 上海前隆信息科技有限公司 | A kind of construction method of data warehouse, system and server |
Cited By (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111680108A (en) * | 2019-03-11 | 2020-09-18 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and data acquisition method and device |
| CN111680108B (en) * | 2019-03-11 | 2023-11-03 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and data acquisition method and device |
| CN111694810B (en) * | 2019-03-12 | 2024-04-05 | 阿里巴巴集团控股有限公司 | Data warehouse creation method and device, electronic equipment and readable storage medium |
| CN111694810A (en) * | 2019-03-12 | 2020-09-22 | 阿里巴巴集团控股有限公司 | Data warehouse creation method and device, electronic equipment and readable storage medium |
| CN110222123A (en) * | 2019-04-24 | 2019-09-10 | 深圳点猫科技有限公司 | The method and electronic equipment that Hive dynamic partition for teaching platform loads |
| CN110119391A (en) * | 2019-05-14 | 2019-08-13 | 重庆八戒传媒有限公司 | A kind of data warehouse creation method and data warehouse based on service data |
| CN110263052A (en) * | 2019-06-25 | 2019-09-20 | 苏宁消费金融有限公司 | One kind automating simultaneous techniques innovative approach based on big data Hadoop platform ODS |
| CN110263052B (en) * | 2019-06-25 | 2021-07-20 | 苏宁消费金融有限公司 | Automatic synchronization technology innovation method based on big data Hadoop platform ODS |
| CN110334088A (en) * | 2019-07-11 | 2019-10-15 | 江苏曲速教育科技有限公司 | Educational data management system |
| CN110850824A (en) * | 2019-11-12 | 2020-02-28 | 北京矿冶科技集团有限公司 | Implementation method for acquiring data of distributed control system to Hadoop platform |
| CN111008234A (en) * | 2019-11-27 | 2020-04-14 | 杭州安恒信息技术股份有限公司 | Warehouse processing method based on network safety data management |
| CN111143465A (en) * | 2019-12-11 | 2020-05-12 | 深圳市中电数通智慧安全科技股份有限公司 | Method and device for realizing data center station and electronic equipment |
| CN111460045A (en) * | 2020-03-02 | 2020-07-28 | 心医国际数字医疗系统(大连)有限公司 | Modeling method, model, computer device and storage medium for data warehouse construction |
| CN111475528A (en) * | 2020-03-23 | 2020-07-31 | 深圳市酷开网络科技有限公司 | OTT-based data warehouse construction method, equipment and storage medium |
| CN111639121A (en) * | 2020-04-07 | 2020-09-08 | 国网新疆电力有限公司 | Big data platform and method for constructing customer portrait |
| CN111461621A (en) * | 2020-04-13 | 2020-07-28 | 郑州工程技术学院 | Distributed school financial management system, method, equipment and storage medium |
| CN111259068A (en) * | 2020-04-28 | 2020-06-09 | 成都四方伟业软件股份有限公司 | Data development method and system based on data warehouse |
| CN112084182A (en) * | 2020-09-10 | 2020-12-15 | 重庆富民银行股份有限公司 | Data modeling method for data mart and data warehouse |
| CN112148807A (en) * | 2020-09-28 | 2020-12-29 | 中国电波传播研究所（中国电子科技集团公司第二十二研究所） | Electromagnetic environment field data warehouse construction method |
| CN112231301A (en) * | 2020-10-21 | 2021-01-15 | 黄河水利委员会黄河水利科学研究院 | Yellow river water sand change data warehouse |
| CN112687097A (en) * | 2020-11-16 | 2021-04-20 | 招商新智科技有限公司 | Highway highway section level data center platform system |
| CN112380218B (en) * | 2020-11-18 | 2023-03-28 | 浪潮通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112380218A (en) * | 2020-11-18 | 2021-02-19 | 浪潮天元通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112860659A (en) * | 2021-01-18 | 2021-05-28 | 北京奇艺世纪科技有限公司 | Data warehouse construction method, device, equipment and storage medium |
| CN112860659B (en) * | 2021-01-18 | 2023-09-01 | 北京奇艺世纪科技有限公司 | Data warehouse construction method, device, equipment and storage medium |
| CN112966024A (en) * | 2021-03-12 | 2021-06-15 | 江苏苏伦大数据科技研究院有限公司 | Financial wind control data analysis system based on big data |
| CN112988919A (en) * | 2021-04-30 | 2021-06-18 | 广东电网有限责任公司 | Power grid data market construction method and system, terminal device and storage medium |
| CN113486096A (en) * | 2021-06-21 | 2021-10-08 | 上海百秋电子商务有限公司 | Multi-library timing execution report data preprocessing and query method and system |
| CN113515362B (en) * | 2021-07-12 | 2023-10-20 | 广州云从洪荒智能科技有限公司 | Data processing method, device, computer equipment and storage medium |
| CN113515362A (en) * | 2021-07-12 | 2021-10-19 | 广州云从洪荒智能科技有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN114385121A (en) * | 2022-01-13 | 2022-04-22 | 浙江工企信息技术股份有限公司 | Software design modeling method and system based on business layering |
| CN114595294A (en) * | 2022-03-11 | 2022-06-07 | 北京梦诚科技有限公司 | Data warehouse modeling and extracting method and system |
| CN114880405A (en) * | 2022-03-31 | 2022-08-09 | 华能信息技术有限公司 | Data lake-based data processing method and system |
| CN115618842A (en) * | 2022-12-15 | 2023-01-17 | 浙江蓝鸽科技有限公司 | Integrated intelligent campus data center system |
| CN116737846A (en) * | 2023-05-31 | 2023-09-12 | 深圳华夏凯词财富管理有限公司 | Asset management data safety protection warehouse system based on Hive |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109189764A (en) | A kind of colleges and universities' data warehouse layered design method based on Hive | |
| Sun et al. | Urban spatial structure and commute duration: An empirical study of China | |
| Zhong et al. | Research on China's tourism: A 35‐year review and authorship analysis | |
| Bhanti et al. | E-governance in higher education: Concept and role of data warehousing techniques | |
| CN109189863A (en) | A method of description things time attribute is simultaneously searched based on the description | |
| Bai et al. | Intelligent platform for real-time page view statistics using educational big data digital resource sharing | |
| Li et al. | Research and analysis of student portrait based on campus big data | |
| Arnaboldi et al. | Studying multicultural diversity of cities and neighborhoods through social media language detection | |
| Hu et al. | Research on smart education service platform based on big data | |
| CN107944845A (en) | A kind of method and device that group's management is carried out by cultural cloud platform | |
| CN114385369A (en) | Traffic transport practitioner education platform based on big data analysis and cloud computing | |
| Zhang | A campus big-data platform architecture for data mining and business intelligence in education institutes | |
| Fadahunsi | A perspective view on the development and applications of Geographical Information System (GIS) in Nigeria | |
| Otcheskiy et al. | Developing tourist destination potential under influence of internal and external factors | |
| Ruoxin et al. | Design of MICE service platform based on big data | |
| Alquier et al. | knowIT, a semantic informatics knowledge management system | |
| Pham et al. | Data warehousing for lifelong learning analytics | |
| Vasilev | Corona Virus Disease 2019 (COVID-19) and its Effect on Cultural Heritage Museums. Comparative Analysis Across Central and Eastern Europe | |
| Yahya Menteşe et al. | A “Resilient Urban Development Decision Support Environment (RUD-DSE)” for Istanbul | |
| Zhang et al. | Discussion on the Wisdom Learning Space and Library Culture Construction in the Information Age | |
| Jie | Hunan Sany Polytechnic College, Changsha 410011, Hunan, China dominic_71268@ 163. com | |
| WIRTHMANN | WP3 Report 4 1 | |
| Yuan et al. | Research and Practice on Campus Big Data Foundation Platform | |
| Cheng et al. | Campus one network management platform based on data service architecture | |
| Wang et al. | Data modeling for the data mart in the prediction system of specialty setting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 1218, 12th floor, building 8, East District, yard 9, Linglong Road, Haidian District, Beijing 100089 Applicant after: BEIJING TAOHUADAO INFORMATION TECHNOLOGY Co.,Ltd. Address before: Room 1503, Yanshan Hotel, No. 38 Guancun Avenue, Haidian District, Beijing Applicant before: BEIJING TAOHUADAO INFORMATION TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190111 |
|
| RJ01 | Rejection of invention patent application after publication |