CN116775948A - Data warehouse system for YMS system and construction method thereof - Google Patents
Data warehouse system for YMS system and construction method thereof Download PDFInfo
- Publication number
- CN116775948A CN116775948A CN202310781378.9A CN202310781378A CN116775948A CN 116775948 A CN116775948 A CN 116775948A CN 202310781378 A CN202310781378 A CN 202310781378A CN 116775948 A CN116775948 A CN 116775948A
- Authority
- CN
- China
- Prior art keywords
- data
- constructing
- data warehouse
- analysis
- warehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title claims abstract description 16
- 238000004458 analytical method Methods 0.000 claims abstract description 36
- 238000007405 data analysis Methods 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 31
- 238000003860 storage Methods 0.000 claims abstract description 29
- 238000013461 design Methods 0.000 claims abstract description 12
- 238000011049 filling Methods 0.000 claims abstract description 12
- 238000004140 cleaning Methods 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 8
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000000034 method Methods 0.000 claims description 13
- 238000005516 engineering process Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 238000013439 planning Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000013500 data storage Methods 0.000 description 9
- 239000008186 active pharmaceutical agent Substances 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000013523 data management Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012098 association analyses Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 241000283973 Oryctolagus cuniculus Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000009960 carding Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000001556 precipitation Methods 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of databases, and discloses a data warehouse system for YMS systems and a construction method thereof, wherein the technical scheme is as follows: comprising the following steps: the data warehouse storage module is used for constructing a data warehouse storage platform with a distributed architecture and combining structuring and unstructured; the data warehouse design module is used for carrying out classified extraction, cleaning and filling processing on the data of each data source, and then establishing various database models; the data query analysis module is used for constructing an independent distributed data analysis software and hardware platform, interfacing with front-end and back-end systems, realizing data analysis processing and feeding back analysis results to a client system slot, and can open structured data and unstructured data to form an integral large data warehouse and a data mart, so that the data query analysis and data association sharing application efficiency is improved.
Description
Technical Field
The present invention relates to the field of databases, and more particularly, to a data warehouse system for a YMS system and a construction method thereof.
Background
The system is a full scale yield management system of YMS, and aims at management improvement of yield of booster products for big data analysis systems of production, process, equipment and quality of general semiconductor manufacturing industry. The system has the industrial characteristics of diversified data analysis objects, extremely large data volume, extremely large data analysis dimension and extremely complex data analysis model, so that the model design of the data warehouse plays a decisive role in the data storage and analysis efficiency of the YMS system. At present, the system in the industry commonly adopts a traditional relational data storage technology and a database model, such as a typical Oracle relational database and a Stored Procedure to execute database establishment and big data statistics.
The key point of the YMS data warehouse is to realize efficient data storage and rapid query analysis on the premise that the data volume of a factory is continuously increased, and the related data volume of YMS produced by a semiconductor industry factory every year can reach hundred million or billion levels (row storage volume of a resume data table), and the defects of the current background technology mainly include the following points:
(1) The large-scale data analysis performance of the factory has a bottleneck, when the analysis data volume reaches TB level or the line storage volume reaches more than ten millions, the data query and analysis performance can be drastically reduced, even the phenomenon of dead halt caused by exhaustion of analysis computer resources of a server or a client can occur, and the use requirements of the large-scale data volume and the complex analysis model of the factory can not be met;
(2) The requirements of correlation analysis of structured data and unstructured data cannot be met;
(3) Efficient association analysis across data warehouse platforms cannot be satisfied;
(4) The data storage and backup security mechanism has high cost;
(5) The data storage expansion cost is high, and the data query analysis efficiency has an upper limit bottleneck after the expansion to a certain volume, and the data query analysis efficiency cannot be matched with the expansion of software and hardware nodes to continuously and linearly increase.
Disclosure of Invention
The invention aims to provide a data warehouse system for a YMS system and a construction method thereof, which can open up structured data and unstructured data to form an integral large data warehouse and a data mart, and improve the efficiency of data query analysis and data association sharing application.
The technical aim of the invention is realized by the following technical scheme: a data warehouse system for a YMS system, comprising:
the data warehouse storage module is used for constructing a data warehouse storage platform with a distributed architecture and combining structuring and unstructured;
the data warehouse design module is used for carrying out classified extraction, cleaning and filling processing on the data of each data source, and then establishing various database models;
the data query analysis module is used for constructing an independent distributed data analysis software and hardware platform, interfacing with front-end and back-end systems, realizing data analysis and processing and feeding back analysis results to the client system;
as a preferable technical scheme of the invention, the data warehouse storage platform comprises a GP database platform and an HDFS distributed file system.
As a preferred technical solution of the present invention, the database module includes data warehouse and data marts of various dimensions and business subjects.
As a preferable technical scheme of the invention, the distributed data analysis software and hardware platform is integrated with Kafka, spark stream processing tools and a memory calculation algorithm library.
A method of constructing a data warehouse system for a YMS system, comprising the steps of:
s1, constructing a data warehouse storage module to form a data warehouse storage platform with a distributed architecture and combining structuring and unstructured;
s2, constructing a data processing sub-module of the data warehouse design module to classify and extract various data sources, and then cleaning and filling the data sources;
s3, constructing a data warehouse model building sub-module of the data warehouse design module to build a database model with dimensions and business subjects;
s4, constructing a data query analysis module and establishing an independent distributed data analysis software and hardware platform.
As a preferred technical solution of the present invention, the data warehouse storage platform that forms a combination of distributed architecture, structured and unstructured, includes:
s1.1, planning hardware specifications of a data warehouse storage platform according to the application scale of data;
s1.2, configuring and adjusting parameters of an operating system;
s1.3, installing data warehouse storage software, wherein the data warehouse storage software comprises distributed architecture database software, structured database software and unstructured database software.
As a preferred solution of the present invention, the classification extraction for the data sources is obtained by timing acquisition of timing tasks.
As a preferred technical solution of the present invention, the establishing of the data warehouse model includes:
constructing a data lake, namely extracting response data from a data source in real time according to different data analysis subjects to form a plurality of data lakes corresponding to the data analysis subjects;
constructing a data warehouse, creating a dimension object table according to the data analysis dimension, and creating a business object table facing a business theme according to the data analysis theme; according to the business scene, extracting and summarizing from a database to obtain a data set corresponding to the data analysis dimension and the data analysis subject field, namely a data warehouse;
constructing a data mart, and extracting the obtained data multidimensional body from the database according to the multidimensional index according to the user requirement, namely the data mart;
and constructing a data center, integrating a data processing technology, and constructing a tool for carrying out standardized processing on the data, namely the data center.
As a preferable technical scheme of the invention, the construction of the data query analysis module comprises the following steps: according to the data analysis theme, a data calculation API is established, a memory calculation technology is adopted, and data calculation preprocessing is executed through an independent distributed data analysis software and hardware platform to provide a data service API.
In summary, the invention has the following beneficial effects: (1) The system can support diversified data sources, can support a structured relational database and unstructured data, and can open the structured data and the unstructured data to form an integral large data warehouse and a data mart;
(2) The method can support the application of the volume of TB-level and PB-level data, can support the second-level query of TB-level and PB-level data, and can meet the data query of the data table row stock of billions;
(3) The data storage can be flexibly and transversely expanded, the mode of transversely expanding GP and Spark distributed computer hardware can be supported, the data query analysis performance can be enhanced along with the increase of transverse expansion nodes, the future data increment requirement can be continuously met, and the data increment processing performance can be expanded.
(4) The data warehouse and the data mart can be established according to the data application service scene, and the data query analysis and the data association sharing application efficiency are improved.
Drawings
FIG. 1 is a schematic diagram of a system of the present invention;
FIG. 2 is a diagram of a common YMS data source object of the present invention;
FIG. 3 is a schematic diagram of dimension and index classification of the present invention;
FIG. 4 is a schematic diagram of a data warehouse model of the present invention;
FIG. 5 is a schematic illustration of the capacity expansion performance increase curve of the present invention;
FIG. 6 is a diagram of an analysis of the construction of a data warehouse/data mart of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
As shown in fig. 1-6, the present invention provides a data warehouse system for YMS system and a construction method thereof, as shown in fig. 1, wherein the data warehouse system comprises: the system comprises a data warehouse storage module, a data warehouse design module and a data query analysis module.
The data warehouse storage module is used for constructing a data warehouse storage platform with a distributed architecture and combining structuring and unstructured; the data warehouse storage platform comprises a GP (Green Plum) database platform and a HDFS (Hadoop Distributed File System) distributed file system, so that the distributed storage of structured and unstructured mass data is satisfied, the traditional YMS data warehouse storage platform architecture is broken, flexible transverse expansion is realized, and no data storage performance bottleneck is caused.
The data warehouse design module is used for carrying out classified extraction, cleaning and filling processing on the data of each data source, and then establishing various database models; the database module comprises data warehouse and data marts with various dimensions and business subjects, so that the data quality and the efficiency of flexible sharing use of data are greatly improved.
The data query analysis module is used for constructing independent distributed data analysis software and hardware platforms outside the YMS client and the server, adopting Kafka and Spark stream processing and memory calculation technology, butting with front and rear end systems, realizing data analysis processing and feeding back analysis results to the client system, and flexibly expanding Spark distributed server nodes after factory productivity improvement and data analysis demand increase, so that performance bottlenecks are avoided;
and a Kafka and Spark stream processing tool and a memory calculation algorithm library are integrated in the distributed data analysis software and hardware platform.
The invention also discloses a corresponding construction method corresponding to the data warehouse system, which comprises the following steps:
s1, constructing a data warehouse storage module to form a data warehouse storage platform with a distributed architecture and combining structuring and unstructured; the method comprises the following specific steps:
s1.1, planning hardware specifications of a data warehouse storage platform according to the application scale of data; for example, 2 Master management nodes and 6 Node data nodes can be built according to the application scale requirements of actual data;
s1.2, configuring and adjusting parameters of an operating system; such as configuration and tuning of firewalls, OS kernel resources, disk I/O, users, etc.
S1.3, installing data warehouse storage software, wherein the data warehouse storage software comprises distributed architecture database software, structured database software and unstructured database software, and comprises the steps of establishing mutual trust and catalogues, checking disk I/O, memory and network performance, initializing a database, configuring Standby configuration, configuring Master Failover and Restoration configuration, configuring a main instance mirror image, configuring environment variables and installing monitoring tools such as Prometheus or Zabbix.
S2, constructing a data processing sub-module of the data warehouse design module to classify and extract various data sources, and then cleaning and filling the data sources;
the high-quality data is the basis of YMS industrial large data warehouse, after the data storage layer is established, operations such as extraction, cleaning, filling and the like are required to be established according to various data sources in a classified manner, so that the integrity and relevance of the data are ensured, and the technical means adopted in the method include File Loader, ETL data extraction task scheduling, rabbit MQ batch processing, storm or Spark stream processing, and the real-time integrity of the data processing is ensured.
The steps of split extraction, cleaning and filling treatment for various data sources are as follows:
s2.1, setting a metadata specification model, such as a data source generation rule, acquisition frequency, data density, parameter quantity, parameter numerical range and the like, according to various data objects.
S2.2, executing a timing data acquisition task, such as an ETL Java script, and automatically acquiring data to the multi-bin platform at fixed time according to the acquisition frequency.
S2.3, executing a data cleaning script, automatically checking whether the data density, the parameter number and the parameter value are matched with the specification definition according to the data specification model, marking the abnormal data logically, and recording the density missing data range abnormally.
S2.4, executing the abnormal data processing script, and eliminating the abnormal data.
S2.5, executing a data filling processing script, and according to the data source production rule, performing association analysis on the reasons of the data source deficiency, if the data source is stopped and the production is stopped, filling is not needed, if the data generation or acquisition process is abnormal, backtracking the data source according to the data generation rule, and generally regenerating the data through software and hardware system log analysis and upstream system association data record, and filling the data into a data warehouse.
S3, constructing a data warehouse model building sub-module of the data warehouse design module to build a database model with dimensions and business subjects;
the data warehouse model establishment comprises the following steps:
constructing a data lake, namely extracting response data from a data source in real time according to different data analysis subjects, and forming a plurality of data lakes corresponding to the data analysis subjects on a GP database platform; a common YMS data source object is shown in fig. 2.
The data lake is a large warehouse (comprising structured and unstructured data) for intensively storing various original data of enterprises, can store the data according to the original state, does not need to define a data model in advance, and has high fidelity, wherein the data can be accessed, processed, analyzed and shared for transmission. The large-scale long-term raw data can be collected, refined, stored and analyzed and mined by utilizing low-cost technology.
Constructing a data warehouse, creating a dimension object table according to the data analysis dimension, and creating a business object table facing a business theme according to the data analysis theme; according to the business scene, extracting and summarizing from a database to obtain a data set corresponding to the data analysis dimension and the data analysis subject field, namely a data warehouse; YMS business topic analysis can efficiently obtain clean, complete, accurate dimensions and topic data, as shown in fig. 6.
A data warehouse is a topic-oriented, integrated, relatively stable, data collection that reflects historical changes for supporting management decisions, and is oriented towards analytical data processing. And the heterogeneous data sources are effectively integrated, recombined according to the theme or service after integration, comprise historical data and are not modified after storage. The data warehouse is enterprise-level and can provide data decision support for the operation of the entire enterprise and various departments.
Constructing a data mart, and extracting the obtained data multidimensional body from the database according to the multidimensional index according to the user requirement, namely the data mart;
the data marts meet the requirements of specific departments or users, are stored in a multidimensional mode, and comprise defining dimensions, indexes to be calculated, layers of the dimensions and the like, and generate a data cube facing the decision analysis requirements; data is extracted from enterprise-wide databases, data warehouses.
A data mart is a miniature data warehouse that typically has less data, fewer subject areas, less historical data, department level, data services for personnel in a local area, finer data dimensions, and more accurate scope. The YMS system will typically build data warehouses and data marts according to the user departments and analysis topic plan, as shown in fig. 3.
And constructing a data center, integrating a data processing technology, and constructing a tool for carrying out standardized processing on the data, namely the data center.
The data center is used for collecting, calculating, storing and processing mass data through a data technology, and unifying standards and calibers. After unifying the data, standard data is formed and then stored to form a large data resource layer, so that efficient service is provided. The service has stronger relevance with business of enterprises, is unique to the enterprises and can be reused, is precipitation of the business and data of the enterprises, and can reduce repeated construction and chimney construction and improve differentiated competitive advantages.
The data center mainly solves the problems of data acquisition, storage, communication and use, and makes all the services data and all the data service.
Corresponding to the YMS data warehouse model, corresponding data analysis query APIs are created for each data dimension object, analysis subject, business function, user and the like, and the data are applied to the technologies of modularization, memory calculation and the like, so that the data multiplexing value and the data query access efficiency are improved. A schematic diagram of the correlation model is shown in fig. 4.
S4, constructing a data query analysis module and establishing an independent distributed data analysis software and hardware platform. The construction of the data query analysis module comprises the following steps: according to the data analysis theme, a data calculation API is established, a memory calculation technology is adopted, data calculation preprocessing is executed through an independent distributed data analysis software and hardware platform, and a data service API is provided.
Also established in the data query analysis module is: the API memory calculation algorithm library can be customized and developed according to the data application service scene, the main purpose is to realize the calculation of standardized big data by adopting distributed memory operation, the bottleneck of the traditional data query analysis performance is broken, and the expansion and promotion of the performance can be realized by transversely and flexibly expanding Spark memory calculation server hardware along with the increase of data volume and non-application, and the expansion of the data volume can be supported infinitely theoretically, and the effect diagram is shown in figure 5.
The data warehouse system and the construction method thereof have the advantages that:
(1) The system can support diversified data sources, can support a structured relational database and unstructured data, and can open the structured data and the unstructured data to form an integral large data warehouse and a data mart;
(2) The method can support the application of the volume of TB-level and PB-level data, can support the second-level query of TB-level and PB-level data, and can meet the data query of the data table row stock of billions;
(3) The data storage can be flexibly and transversely expanded, the mode of transversely expanding GP and Spark distributed computer hardware can be supported, the data query analysis performance can be enhanced along with the increase of transverse expansion nodes, the future data increment requirement can be continuously met, and the data increment processing performance can be expanded.
(4) The data warehouse and the data mart can be established according to the data application service scene, and the data query analysis and the data association sharing application efficiency are improved.
The construction method actually implemented by the invention comprises the following steps:
carding service scenes: how the data generates value to the service is clarified;
data priority definition: defining priority class execution according to the data value and the business demand priority;
data management: the IT technical team cooperates with business users to participate in data management, establishes a data asset catalog, combines business scenes, refines a business model, and further establishes a data lake, a data warehouse and a data mart;
data quality: establishing a data quality monitoring and processing flow mechanism, continuously monitoring and improving the data quality, and reversely pushing service informatization, service flow and service data management improvement according to the data quality problem;
data warehouse application API establishment: a standard data access interface is established.
The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the present invention may occur to one skilled in the art without departing from the principles of the present invention and are intended to be within the scope of the present invention.
Claims (9)
1. A data warehouse system for a YMS system, characterized by: comprising the following steps:
the data warehouse storage module is used for constructing a data warehouse storage platform with a distributed architecture and combining structuring and unstructured;
the data warehouse design module is used for carrying out classified extraction, cleaning and filling processing on the data of each data source, and then establishing various database models;
the data query analysis module is used for constructing an independent distributed data analysis software and hardware platform, interfacing with the front-end system and the back-end system, realizing data analysis and processing and feeding back analysis results to the client system.
2. A data warehouse system for a YMS system according to claim 1, characterized in that: the data warehouse storage platform comprises a GP database platform and an HDFS distributed file system.
3. A data warehouse system for a YMS system according to claim 2, characterized in that: the database module includes data warehouses and data marts of various dimensions and business topics.
4. A data warehouse system for a YMS system according to claim 3, characterized in that: and a Kafka, spark stream processing tool and a memory calculation algorithm library are integrated in the distributed data analysis software and hardware platform.
5. A construction method of a data warehouse system for a YMS system is characterized in that: the method comprises the following steps:
s1, constructing a data warehouse storage module to form a data warehouse storage platform with a distributed architecture and combining structuring and unstructured;
s2, constructing a data processing sub-module of the data warehouse design module to classify and extract various data sources, and then cleaning and filling the data sources;
s3, constructing a data warehouse model building sub-module of the data warehouse design module to build a database model with dimensions and business subjects;
s4, constructing a data query analysis module and establishing an independent distributed data analysis software and hardware platform.
6. A method of constructing a data warehouse system for a YMS system according to claim 5, wherein: the data warehouse storage platform forming a combination of distributed architecture, structured and unstructured, comprises:
s1.1, planning hardware specifications of a data warehouse storage platform according to the application scale of data;
s1.2, configuring and adjusting parameters of an operating system;
s1.3, installing data warehouse storage software, wherein the data warehouse storage software comprises distributed architecture database software, structured database software and unstructured database software.
7. A method of constructing a data warehouse system for a YMS system according to claim 6, wherein: the categorical extraction of data sources is obtained by timed acquisition of timed tasks.
8. A method of constructing a data warehouse system for a YMS system according to claim 7, wherein: the data warehouse model establishment comprises the following steps:
constructing a data lake, namely extracting response data from a data source in real time according to different data analysis subjects to form a plurality of data lakes corresponding to the data analysis subjects;
constructing a data warehouse, creating a dimension object table according to the data analysis dimension, and creating a business object table facing a business theme according to the data analysis theme; according to the business scene, extracting and summarizing from a database to obtain a data set corresponding to the data analysis dimension and the data analysis subject field, namely a data warehouse;
constructing a data mart, and extracting the obtained data multidimensional body from the database according to the multidimensional index according to the user requirement, namely the data mart;
and constructing a data center, integrating a data processing technology, and constructing a tool for carrying out standardized processing on the data, namely the data center.
9. A method of constructing a data warehouse system for a YMS system according to claim 8, wherein: the construction of the data query analysis module comprises the following steps: according to the data analysis theme, a data calculation API is established, a memory calculation technology is adopted, and data calculation preprocessing is executed through an independent distributed data analysis software and hardware platform to provide a data service API.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310781378.9A CN116775948A (en) | 2023-06-29 | 2023-06-29 | Data warehouse system for YMS system and construction method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310781378.9A CN116775948A (en) | 2023-06-29 | 2023-06-29 | Data warehouse system for YMS system and construction method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116775948A true CN116775948A (en) | 2023-09-19 |
Family
ID=88009699
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310781378.9A Pending CN116775948A (en) | 2023-06-29 | 2023-06-29 | Data warehouse system for YMS system and construction method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116775948A (en) |
-
2023
- 2023-06-29 CN CN202310781378.9A patent/CN116775948A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Santos et al. | Real-time data warehouse loading methodology | |
| CN103425734A (en) | Database, apparatus, and method for storing encoded triples | |
| CN111930768B (en) | Incremental data acquisition method, incremental data transmission method, incremental data acquisition device, incremental data transmission device and computer storage medium | |
| CN109669975B (en) | Industrial big data processing system and method | |
| Tank et al. | Speeding ETL processing in data warehouses using high-performance joins for changed data capture (cdc) | |
| CN111126852A (en) | BI application system based on big data modeling | |
| CN115858513A (en) | Data governance method, data governance device, computer equipment and storage medium | |
| CN112100227A (en) | Big data processing method based on multilevel heterogeneous data storage | |
| US20180181621A1 (en) | Multi-level reservoir sampling over distributed databases and distributed streams | |
| Yao et al. | Relational Database Query Optimization Strategy Based on Industrial Internet Situation Awareness System | |
| CN114691762A (en) | Intelligent construction method for enterprise data | |
| CN117171135A (en) | User behavior analysis modeling method, analysis method and system | |
| KR101331350B1 (en) | Large-scale, time-series data handling method using data cube | |
| CN116795816A (en) | Stream processing-based multi-bin construction method and system | |
| CN111046054A (en) | Method and system for analyzing power marketing business data | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| CN116775948A (en) | Data warehouse system for YMS system and construction method thereof | |
| Munirathinam et al. | Design and implementation of manufacturing data lake in Hadoop | |
| CN115599871A (en) | Lake and bin integrated data processing system and method | |
| SABTU et al. | THE CHALLENGES OF EXTRACT, TRANSFORM AND LOAD (ETL) FOR DATA INTEGRATION IN NEAR REALTIME ENVIRONMENT. | |
| CN112101894A (en) | Coal dressing intelligent system | |
| CN111984756A (en) | Method for realizing Hbase secondary index | |
| Sarr et al. | Data stream summary in big data context: challenges and opportunities | |
| Tank | Reducing ETL load times by a new data integration approach for real-time business intelligence | |
| CN114168612B (en) | Asset big data platform query acceleration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |