CN116795816A - A data warehouse construction method and system based on streaming processing - Google Patents

Abstract

The application discloses a stream processing-based method and a stream processing-based system for constructing a plurality of bins, which comprise the steps of analyzing and restoring service data of a structured or unstructured standard data packet, monitoring and capturing the change of a database, analyzing and processing the change, and pushing the data to a data convergence layer ODS; the data convergence layer ODS cleans, converts and desensitizes the data, and associates the data to form a data detail layer DWD; the data in the data detail layer DWD is distributed through the data to form a summarized data layer DWS, or the data is synchronized to a cloud component according to service requirements to form a standardized data query service; and the summary data layer DWS distributes the data detail layer DWD into a wide table or a thematic library through MYLink SQL data, and pi outputs the calculated data to the cloud component to provide service inquiry and offline calculation analysis. The stream processing-based multi-bin construction method and system have strong adaptability to scenes with high data real-time requirements, can be rapidly deployed and are easy to maintain, and the cost of enterprises is greatly reduced and the adaptability is improved.

Description

A data warehouse construction method and system based on streaming processing

Technical field

The invention relates to the technical field of data warehouse construction, in particular to a data warehouse construction method and system based on stream processing.

Background technique

In the computer field, data warehouse is a system used for data analysis and reporting, and is an important component of business intelligence. A data warehouse is a central repository of integrated data from one or more disparate sources, which can be heterogeneous. Data warehouses generally store the latest or historical data together for the creation of business value. Data warehouse represents the way of managing and using data. It is a typical complete system of extraction, transformation, loading, modeling, usage classification, data integration and access layer. A data warehouse is a subject-oriented, integrated, changing, but relatively stable data collection used to support the management decision-making process.

In the Internet era, due to the rapid increase in Internet users, especially from the PC era to the mobile Internet era, massive user behavior data are generated, from PB level to EB level, or even ZB level every day. Enterprises are eager to mine effective results from these massive data. Extract valuable information from business information, such as business data, user behavior data, abnormal data, etc., for users to make business decisions. As the scale of data continues to increase, the amount of business data surges, and the construction methods and architecture of data warehouses are constantly iterated, from the original traditional data warehouse to offline data warehouse, and now to offline plus real-time Lambda.

Traditional data warehouses regularly load structured, semi-structured and unstructured data from source data into the data warehouse through offline ETL, and then obtain the results through the calculation engine, and then provide the front-end or service for use. Offline data warehouse + computing engine is usually used in large-scale OLTP databases (mainly traditional relational databases, such as Oracle and SQL Server). Traditional data warehouses can perfectly support transactional business processing, but they are unable to adapt to massive data storage and calculations. The main reasons are as follows:

1. The traditional data warehouse is a pre-built model, which requires constant reconstruction for massive data or changing user needs, which is very inefficient.

2. Since the massive data of big data changes from quantitative to qualitative changes, traditional data warehouses cannot meet the needs of rapid analysis and response to massive data.

3. The cluster expansion cost of traditional data warehouse is very high, and it is difficult to do horizontal expansion to improve computing power.

As the scale of data continues to increase, the amount of business data surges. Traditional data warehouses are unable to carry massive amounts of data. Traditional database storage technologies are also facing storage shortages and increasing costs. At the same time, with the popularization of big data technology, it is possible to build offline data warehouses through big data technology, and use big data technology to store and calculate offline data warehouses. The construction of data warehouse in big data is based on the traditional data warehouse construction architecture. Big data technical tools are used to replace traditional OLTP, and it evolves into the proposed offline big data data warehouse architecture. The offline data warehouse construction method solves the problem well. The shortcomings of traditional data warehouses. As business data processing capabilities and needs continue to change, it has been found in practice that although the offline batch processing mode capabilities have been greatly improved, it cannot satisfy business scenarios with very high data processing and business timeliness requirements. The "offline + streaming computing" dual-link data warehouse construction method is a compromise transitional solution, but it has many shortcomings in practical production:

1. Increased usage of computing resources. Since there are two lines of offline and streaming computing at the same time, the time periods occupied by offline and streaming data computing resources may be inconsistent. Offline computing is more likely to occur between 12 a.m. and before 6 a.m., while streaming computing is more likely to occur during the day or during the day. Before 12 o'clock in the morning, such offline and streaming computing resources cannot be fully utilized, resulting in an increase in overall resource usage.

2. Maintain two sets of codes at the same time. For the two lines of offline and streaming computing, one needs to implement the code on the offline engine, and the other needs to implement the code on the streaming engine, and two sets of test processes need to be implemented. The operation and maintenance costs of the warehouse business have doubled.

3. Offline calculation has poor timeliness. Due to the continuous changes in business, more and more businesses need to have higher and higher timeliness requirements for the original offline tasks. Since offline computing can only meet the calculation requirements of T+n, the time level of n can only be transferred to minutes. level, which places higher and higher demands on server resources, and inconsistent timeliness can be guaranteed.

4. Cluster storage requirements are high. Since both offline and streaming link processes require data to be stored in the cluster, and a large amount of temporary data or logs will be generated during the intermediate calculation process, this will cause data to expand rapidly and put great pressure on server storage.

Contents of the invention

In order to solve the above technical problems existing in the prior art, the present invention proposes a data warehouse construction method and system based on flow processing to solve the above technical problems.

According to the first aspect of the present invention, a data warehouse construction method based on streaming processing is proposed, including:

S1: Analyze and restore business data for structured or unstructured standard data packages, monitor and capture changes in the database for analysis and processing, and push the data to the data aggregation layer ODS;

S2: The data aggregation layer ODS cleans, converts, desensitizes, and associates the data to form the data detail layer DWD;

S3: The data in the data detail layer DWD forms the summary data layer DWS through data distribution, or the data is synchronized to the cloud component according to business needs to form a standardized data query service;

S4: The summary data layer DWS distributes the data detail layer DWD through MYLink SQL to form a wide table or thematic library, and outputs the calculated data to the cloud component to provide service query and offline calculation analysis.

In some specific embodiments, S1 also includes collecting data from the source business library according to the collection rules of the data aggregation layer ODS. The source business library can correspond to one or more data aggregation layer ODS.

In some specific embodiments, S1 specifically includes using the sSend tool to parse structured or unstructured standard data packets for business data, using Datax to parse and restore structured or unstructured data, and using FlinkCDC to monitor and capture the database. Changes are analyzed and processed.

In some specific embodiments, the business database data stored by the data aggregation layer ODS in S2 maintains the original appearance of the business data, and the MYLink engine uses SQL+UDF to clean, convert, desensitize, and associate the data to form data details. Layer DWD.

In some specific embodiments, S2 also includes outputting the data directly to the cloud component through the MYLink engine to provide tracking queries for the original data.

In some specific embodiments, the cloud components include Huawei Cloud certified components or Tencent Cloud.

In some specific embodiments, the sSend tool, Datax, and FlinkCDC all support consumption queues as data aggregation layer ODS.

According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above-mentioned method is implemented.

According to the third aspect of the present invention, a data warehouse construction system based on stream processing is proposed, including:

The data processing unit is configured to parse and restore structured or unstructured standard data packets, monitor and capture changes in the database for parsing and processing, and push the data to the data aggregation layer ODS. The data aggregation layer ODS processes the data. Perform cleaning, conversion, desensitization, and correlation to form the data detail layer DWD;

The data distribution unit is configured to distribute the data in the detailed data layer DWD to form the summary data layer DWS, or synchronize the data to the cloud component according to business needs to form a standardized data query service;

The query analysis unit, the summary data layer DWS, distributes the data detail layer DWD through MYLink SQL to form a wide table or thematic library, and outputs the calculated data to the cloud component to provide service query and offline calculation analysis.

In some specific embodiments, it also includes a data collection unit configured to collect data from the source business library according to the collection rules of the data aggregation layer ODS. The source business library can correspond to one or more data aggregation layer ODS.

In some specific embodiments, the sSend tool is used to parse structured or unstructured standard data packets for business data, Datax is used to parse and restore structured or unstructured data, and FlinkCDC is used to monitor and capture database changes. Parsing and processing; the data aggregation layer ODS is used to store business database data to keep the original appearance of the business data, and use the MYLink engine to clean, convert, desensitize, and associate the data in the form of SQL+UDF to form the data detail layer DWD; use the MYLink engine to The data is directly output to the cloud component to provide tracking queries of the original data.

In some specific embodiments, the sSend tool, Datax, and FlinkCDC all support consumption queues as data aggregation layer ODS.

The present invention proposes a data warehouse construction method and system based on streaming processing, which can be well adapted to business scenarios where streams and batches coexist. The streams and batches have the same set of codes and can share the same resources, which improves resource utilization. High and low resource overhead. As long as a set of code is implemented, the difficulty of development, testing, release and online is greatly reduced, and the later operation and maintenance costs are also reduced. It has strong adaptability to scenarios that require high real-time data performance. The advantages of rapid deployment and easy maintenance can greatly reduce the cost of enterprises and improve adaptability.

Description of the drawings

The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and together with the description serve to explain the principles of the invention. Other embodiments and many of the intended advantages of embodiments will be readily recognized as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

Figure 1 is a flow chart of a data warehouse construction method based on streaming processing according to an embodiment of the present application;

Figure 2 is an overall architecture diagram of a streaming data warehouse according to a specific embodiment of the present application;

Figure 3 is a flow chart of data warehouse construction based on streaming processing according to a specific embodiment of the present application;

Figure 4 is a data warehouse architecture diagram of a specific embodiment of the present application;

Figure 5 is an architecture diagram of a data warehouse construction system based on streaming processing according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.

Detailed ways

The present application will be further described in detail below in conjunction with the accompanying drawings and examples. It can be understood that the specific embodiments described here are only used to explain the relevant invention, but not to limit the invention. It should also be noted that, for convenience of description, only the parts related to the invention are shown in the drawings.

It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments of this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

Figure 1 shows a flow chart of a data warehouse construction method based on streaming processing according to an embodiment of the present application. As shown in Figure 1, the method includes the following steps:

S101: Analyze and restore business data for structured or unstructured standard data packages, monitor and capture changes in the database for analysis and processing, and push the data to the data aggregation layer ODS.

In a specific embodiment, S1 also includes collecting data from the source business library according to the collection rules of the data aggregation layer ODS. The source business library can correspond to one or more data aggregation layer ODS. Use the sSend tool to parse structured or unstructured standard data packets for business data, use Datax to parse and restore structured or unstructured data, and use FlinkCDC to monitor and capture changes in the database for analysis and processing.

S102: The data aggregation layer ODS cleans, converts, desensitizes, and correlates the data to form the data detail layer DWD. The data aggregation layer ODS is used to store business database data to keep the original appearance of the business data. The MYLink engine uses SQL+UDF to clean, convert, desensitize and associate the data to form the data detail layer DWD. Data can also be output directly to the cloud component through the MYLink engine to provide tracking and querying of original data. Among them, sSend tool, Datax, and FlinkCDC all support consumption queues as data aggregation layer ODS.

S103: The data in the data detail layer DWD forms the summary data layer DWS through data distribution, or the data is synchronized to the cloud component according to business needs to form a standardized data query service.

S104: The summary data layer DWS distributes the data detail layer DWD through MYLink SQL to form a wide table or thematic library, and outputs the calculated data to the cloud component to provide service query and offline calculation analysis.

Figure 2 shows the overall architecture diagram of a streaming data warehouse according to a specific embodiment of the present application. The streaming data warehouse is based on the new generation streaming computing engine MYLINK as the base, with TJPlat as the workbench, and TJPlat as its base. The core idea is to solve the problems in the data warehouse construction process by integrating and transforming the flow computing system, so that both real-time computing and batch processing can be constructed according to the same model. TJPlat can also recalculate historical data when necessary, reflecting the flexibility of the streaming data warehouse.

In a specific embodiment, the streaming data warehouse construction of this application is based on the top-level construction idea of "four modernizations" of "all resources, resource cataloging, catalog globalization, and global standardization", and the data warehouse construction is hierarchical. Integrated into the data warehouse construction method, all data warehouse hierarchical resources have been defined as all resources; all resources are defined in the resource directory for resource management, data item management, resource classification and classification management, etc., and are also defined or designed in the resource directory The downstream resources or extraction rules extracted by this resource, data cleaning rules (filtering, deduplication, formatting, verification, etc.), data association (Join operation of resources), etc.; the directory globally defines the deployment and storage components of the data warehouse. (Can be ES, Hbase, MongoDB, HDFS, ClickHouse, etc.); Global standardization is to provide global guidance through unified standards for subsequent resource service queries, data applications and even BI analysis; "Four modernizations" are streaming data processing Cang's suggestions provide specific business guidance, which can also be said to be the process of business from abstraction to concreteness.

In a specific embodiment, according to the ODS (aggregation library) collection rules defined in the "Four Modernizations", from the source business library to the aggregation library (ODS), the source library to the aggregation library can be one source library to n (one or Multiple) aggregation libraries, through the open source MDatax, data can be collected (business rules for collection can be defined during the collection process) into offline libraries or consumption queues (Kafka); sSend can collect structured data and semi-structured data of specific business scenarios Data and even unstructured data are collected into offline libraries, object storage or consumption queues, which are suitable for the collection of data in specific scenarios of big data; FlinkCDC is mainly used to obtain real-time data change acquisition (Change DataCapture) from major databases such as MySQL, Oracle, and Postgres. ), the principle is to monitor and capture database changes (database insertion, update and deletion), completely record the sequence of these changes and write them to the consumption queue (Kafka) for MYLINK subscription and consumption. All data collection is controlled based on definable ideas. Datax, sSend or FlinkCDC must support consumption queue (Kafka) as ODS, which is a necessary prerequisite for the construction of streaming data warehouse.

In a specific embodiment, MYLINK is a new generation of streaming computing engine developed based on Flink's batch and stream thinking. It is a streaming computing engine that completely uses SQL-like semantics combined with the "four modernizations" concept of this application, reducing the The threshold for building a streaming data warehouse. MYLINK retains the real-time computing features of Flink, including: supporting both unbounded and unbounded data sources; supporting Join and Union; low latency; and customizable unified connectors.

MYLINK also combines the concept of "four modernizations" to innovate the streaming data warehouse construction model, which is specifically reflected in the following aspects:

(1) Connect the relationship between the ODS layer and the DWD layer through resource benchmarking. MYLINK will perform stream computing processing (such as data cleaning, data association, data extraction and distribution, etc. ETL processes) according to the data processing rules defined by the benchmarked resources. Automatically output hierarchical output of parallel streaming output data according to deployment definitions;

(2) The order and position of each operator link in streaming computing can be dynamically and freely adjusted according to business needs to meet business needs;

(3) In the streaming computing section, you can write your own SQL statements to handle specific business needs, and you can define UDF methods for special or complex businesses;

(4) MYLINK provides a dynamic adjustment "window" method to allocate business batch calculation windows;

(5) Supports more data warehouse components. In addition to supporting open source components, it also supports cloud components from some manufacturers, such as Huawei Cloud certified components (ES, HIVE, HDFS, Mongodb, etc.), Tencent Cloud (ES, HIVE, HDFS, etc.) ).

The data warehouse construction flow chart based on streaming processing according to a specific embodiment of the present application is shown in conjunction with Figure 3, which specifically includes the following steps:

The platform uses the "sSend" tool to parse structured or unstructured standard data packages for business data. The "MDatax" capability is used to parse and restore structured or unstructured data. The "data collection" FlinkCDC is dedicated to Monitor and capture changes in the database for analysis and processing, and push the data to the ODS layer library.

The ODS layer (original library) is used to store business database data to keep the original appearance of the business data. The data in the ODS layer may be cleaned, converted, desensitized, associated, etc. through the MYLink engine in the form of SQL+UDF to form DWD. Through MYLink, data can also be directly output to ES, HDFS and other components for data analysts to track and query the original data. It can also be provided to "data applications" or third-party services through the "cxLevelS" or "Tianhe" service platform.

The data in the DWD layer (data detail layer) is a managed data layer. The data in the DWD layer can be used for "data distribution" to form DWS such as user behavior, product attributes or topic libraries; or the data can be passed through according to business needs. MYLink synchronizes to ES to provide data detail layer query services to form various standardized data query services.

The DWS layer comes from the DWD layer to form a wide table or thematic library (indicator library) through the "data distribution" of MYLink SQL, and outputs the calculated data to Mongo, HDFS and other components to provide service query and offline calculation analysis respectively.

In specific embodiments, the offline data warehouse construction method can well solve the shortcomings of traditional data warehouses: 1. Batch data calculation can largely solve the problem of traditional data warehouses being unable to calculate massive data. . And after batch calculation, the calculated results can be queried in batches. 2. Low-cost computing power expansion. Since the offline data warehouse uses the Hdoop ecosystem, it can perform well on low-cost minicomputers and provides a low-cost expansion solution for horizontal expansion of offline computing. Unlike traditional data warehouses, you can only purchase large machines for vertical expansion. 3. Offline batch calculation can also be based on the actual requirements of the business and unified calculation after the data is in place, which ensures the consistency and integrity of the data. The offline data warehouse uses open source Datax, Flume or the self-developed sSend tool to uniformly collect different sources of heterogeneous data sources into offline storage HDFS or HIVE, and may also collect important basic data into relational databases, etc. The collected data forms a collection ledger log. The data collected in the offline database can be stored in partitions, tables or databases according to resource conditions, and then the collected data can be calculated offline in batches through HiveQL or SparkSQL of QBI (self-developed offline analysis platform), which can be executed in one go or Timed execution. After outputting business data according to business needs, data services can be provided through the unified publishing platform CXLeveS for use by data applications or service applications.

During the construction of the offline data warehouse, it can carry out hierarchical construction and management according to the construction of the data warehouse, so that the data can be circulated in an orderly manner according to the design. Data warehouse stratification provides a very important theoretical basis for data stratification recommendation management. In the offline data warehouse recommendation process, data (tables or resources) rely on complex, hierarchical, and even cyclically dependent data systems. In order to implement the data systematization in an orderly manner, a modern data layering system is needed for effective data organization, management and use. Data stratification is mainly for clearer control of data during the data management process. It has the following advantages:

1. Clear data structure. Each data layer has a corresponding scope, which is more convenient to define and understand when expressing the layer.

2. Simplify complex problems. Each complex task is broken down into multiple steps to complete. Each layer only handles specific problems, which is relatively simple and easy to understand. Easily maintain data accuracy and consistency.

3. Reduce duplication of work. Standardized data stratification can reduce a lot of repeated calculations when developing some common middle-tier data tables.

4. Unified data caliber. Through data layering, a unified data export can be provided, and the caliber of externally output business data can be unified to avoid business abnormalities.

5. Track data sources. The business table is externally unique, but the source of the business table can be one or more, and the data source can be quickly defined by tracking the lineage of the data table.

Figure 4 shows the data warehouse architecture diagram of a specific embodiment of the present application. As shown in Figure 4, the data warehouse is generally divided into three layers, namely data aggregation layer (ODS), data warehouse layer (DW), and data application layer. Layer (ADS), of which the data warehouse layer (DW) can be divided into: data detail layer (DWD), summary data layer (DWS), common dimension layer (DIM), and TMP (temporary data layer).

This application proposes a new data warehouse suggestion method to address many problems in the construction and production of traditional data warehouses and offline data warehouses (including "offline + streaming computing" data warehouses). This data warehouse construction method based on streaming processing can also be called a data warehouse construction method and system based on streaming and batch integration. The solution to building a data warehouse through the integration of flow and batch has the following advantages:

1. Calculate homology. Using the same set of code and the same set of logic can process streaming tasks or batch tasks at the same time, reducing learning and maintenance costs and achieving better resource utilization.

2. Store the same origin. Streaming processing integration can satisfy the storage of streaming data and batch data at the same time on the storage system, and can effectively collaborate and update metadata.

3. Low data latency. Streaming processing is created for real-time. The data delay is at the second or even millisecond level, which is very suitable for business scenarios with demanding real-time requirements.

Figure 5 shows the architecture diagram of a data warehouse construction system based on streaming processing according to one embodiment of the present application. As shown in Figure 5, the system includes a data processing unit 501, a data distribution unit 502 and a query analysis unit 503, where, The data processing unit 501 is configured to analyze and restore business data for structured or unstructured standard data packets, monitor and capture changes in the database for analysis and processing, and push the data to the data aggregation layer ODS. The data aggregation layer ODS processes the data. Perform cleaning, conversion, desensitization, and association to form the data detail layer DWD; the data distribution unit 502 is configured to distribute the data in the data detail layer DWD to form the summary data layer DWS, or synchronize the data to the cloud component according to business needs. Standardized data query service; the summary data layer DWS in the query analysis unit 503 distributes the data detail layer DWD through MYLink SQL to form a wide table or thematic library, and outputs the calculated data to the cloud component to provide service query and offline calculation analysis. .

In a specific embodiment, the data processing unit 501 also includes a data collection unit configured to collect data from the source business library according to the collection rules of the data aggregation layer ODS. The source business library can correspond to one or more data aggregation layer ODS. .

The data warehouse construction method and system based on streaming processing of this application can be well adapted to business scenarios where streams and batches coexist. For streams and batches, the same set of codes can share the same resources. For high resource utilization and resource overhead, Small. As long as a set of code is implemented, the difficulty of development, testing, release and online is greatly reduced, and the later operation and maintenance costs are also reduced. Streaming processing has great application scenarios for current time-sensitive business needs, such as fraud prediction. Fraud is a high-incidence industry in the financial field. For these frauds, the process of occurrence is short and the impact is large. How to prevent them has been unclear in recent years. There are fewer problems that financial companies or banks need to solve together. Traditional anti-fraud methods are no longer sufficient to solve the difficulties faced. In the past, it took several hours to calculate transaction data and user behavior indicators, identify suspicious users through corresponding rules, and then combine them with case investigation and screening. In this case, the funds have already been transferred by criminals several times on the earth. All over. Using the streaming data warehouse construction method, streaming calculations on the corresponding data can complete the calculation of the corresponding indicators within seconds or even milliseconds, and then provide real-time warning or interception of the real-time flow to avoid losses. The present invention has strong adaptability to scenarios where it is necessary to construct a streaming data warehouse and where real-time data requirements are high. The advantages of rapid deployment and easy maintenance can greatly reduce the cost of enterprises and improve adaptability. .

Referring now to FIG. 6 , which shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. The electronic device shown in FIG. 6 is only an example and should not impose any restrictions on the functions and usage scope of the embodiments of the present application.

As shown in FIG. 6, the computer system includes a central processing unit (CPU) 601, which can operate according to a program stored in a read-only memory (ROM) 602 or loaded from a storage portion 608 into a random access memory (RAM) 603. Perform various appropriate actions and processing. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a liquid crystal display (LCD), etc., speakers, etc.; a storage section 608 including a hard disk, etc.; and including a LAN card, Communication section 609 of a network interface card such as a modem. The communication section 609 performs communication processing via a network such as the Internet. Driver 610 is also connected to I/O interface 605 as needed. Removable media 611, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method illustrated in the flowchart. In such embodiments, the computer program may be downloaded and installed from the network via communication portion 609, and/or installed from removable media 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are performed. It should be noted that the computer-readable storage medium of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmed read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. As used herein, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that may be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device program of. Program code embodied on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

Computer program code for performing the operations of the present application may be written in one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedures, or a combination thereof. programming language - such as "C" or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through Internet connection).

The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.

The modules involved in the embodiments described in this application can be implemented in software or hardware.

As another aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be included in the electronic device described in the above embodiments; it may also exist independently without being assembled into the electronic device. in electronic equipment. The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: parses and restores business data of structured or unstructured standard data packets. , monitor and capture changes in the database for analysis and processing, and push the data to the data aggregation layer ODS; the data aggregation layer ODS cleans, converts, desensitizes, and associates the data to form the data detail layer DWD; the data in the data detail layer DWD passes through the data Distribute to form a summary data layer DWS, or synchronize data to cloud components according to business needs to form a standardized data query service; the summary data layer DWS distributes the data detail layer DWD through MYLink SQL data to form a wide table or thematic library, and the calculated The data is output to the cloud component to provide service query and offline computing analysis.

The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by a specific combination of the above technical features, but should also cover any solution consisting of the above technical features or without departing from the above inventive concept. Other technical solutions formed by any combination of equivalent features. For example, a technical solution is formed by replacing the above features with technical features with similar functions disclosed in this application (but not limited to).

Claims (12)

1. A data warehouse construction method based on streaming processing, which is characterized by including: S1: Analyze and restore business data for structured or unstructured standard data packages, monitor and capture changes in the database for analysis and processing, and push the data to the data aggregation layer ODS; S2: The data aggregation layer ODS cleans, converts, desensitizes, and associates the data to form the data detail layer DWD; S3: The data in the detailed data layer DWD forms a summary data layer DWS through data distribution, or the data is synchronized to the cloud component according to business needs to form a standardized data query service; S4: The summary data layer DWS distributes the data detail layer DWD through MYLink SQL to form a wide table or thematic library, and outputs the calculated data to the cloud component to provide service query and offline calculation analysis. 2. The data warehouse construction method based on streaming processing according to claim 1, characterized in that, before S1, it also includes collecting data from the source business library according to the collection rules of the data aggregation layer ODS. The service library may correspond to one or more of the data aggregation layer ODS. 3. The data warehouse construction method based on streaming processing according to claim 1, characterized in that the S1 specifically includes using the sSend tool to perform business data analysis on the structured or unstructured standard data packets. Datax parses and restores the structured or unstructured data, and uses FlinkCDC to monitor and capture changes in the database for analysis and processing. 4. The data warehouse construction method based on streaming processing according to claim 1, characterized in that, in the S2, the business library data used to store by the data aggregation layer ODS maintains the original appearance of the business data, and uses the MYLink engine to The data is cleaned, converted, desensitized, and associated using SQL+UDF to form the data detail layer DWD. 5. The data warehouse construction method based on streaming processing according to claim 4, characterized in that the S2 further includes directly outputting the data to the cloud component through the MYLink engine to provide tracking query of the original data. 6. The data warehouse construction method based on streaming processing according to claim 1, characterized in that the cloud component includes a Huawei Cloud certification component or Tencent Cloud. 7. The data warehouse construction method based on streaming processing according to claim 3, characterized in that the sSend tool, Datax, and FlinkCDC all support consumption queues as the data aggregation layer ODS. 8. A computer-readable storage medium on which one or more computer programs are stored, characterized in that when the one or more computer programs are executed by a computer processor, the method of any one of claims 1-7 is implemented. method. 9. A data warehouse construction system based on streaming processing, which is characterized by including: The data processing unit is configured to analyze and restore business data for structured or unstructured standard data packets, monitor and capture changes in the database for analysis and processing, and push the data to the data aggregation layer ODS. The data aggregation layer ODS Clean, convert, desensitize, and associate the data to form the data detail layer DWD; A data distribution unit configured to distribute the data in the detailed data layer DWD to form a summary data layer DWS, or synchronize the data to the cloud component according to business needs to form a standardized data query service; Query analysis unit, the summary data layer DWS distributes the data detail layer DWD through MYLink SQL to form a wide table or thematic library, and outputs the calculated data to the cloud component to provide service query and offline calculation analysis. 10. The data warehouse construction system based on streaming processing according to claim 9, characterized in that it also includes a data collection unit configured to collect data from the source business library according to the collection rules of the data aggregation layer ODS, The source service library may correspond to one or more of the data aggregation layer ODS. 11. The data warehouse construction system based on streaming processing according to claim 9, characterized in that the sSend tool is used to analyze the structured or unstructured standard data packets for business data, and Datax is used to analyze the structured data. Parse and restore structured or unstructured data, use FlinkCDC to monitor and capture changes in the database for analysis and processing; the data aggregation layer ODS is used to store the business database data to maintain the original appearance of the business data, using the MYLink engine in the form of SQL+UDF The data is cleaned, converted, desensitized, and associated to form the data detail layer DWD; the data is directly output to the cloud component through the MYLink engine to provide tracking and querying of the original data. 12. The data warehouse construction system based on streaming processing according to claim 11, characterized in that the sSend tool, Datax, and FlinkCDC all support consumption queues as the data aggregation layer ODS.