CN114490610A - Data processing method and device for data bin, storage medium and electronic device - Google Patents

Abstract

The application discloses a data processing method and device for a data bin, a storage medium and an electronic device. The method comprises the steps of acquiring data from different data sources based on a Flink frame, and storing the data according to data lake hudi; establishing a target data bin based on preset data extraction, conversion and loading processing; executing preset data processing operation based on the target data bin, wherein the preset data processing operation at least comprises one of the following operations: generating report operation, business data query operation and decision engine original data cleaning operation. The method and the device solve the technical problems of poor storage and calculation processing effects of the internet financial data. The method and the device for realizing the light-weight deployment and supporting the visual configuration under different service scenes are realized.

Description

Data processing method and device, storage medium and electronic device for data warehouse

technical field

The present application relates to the technical field of data warehouses, and in particular, to a data processing method and device, a storage medium, and an electronic device for a data warehouse.

Background technique

Internet finance, by applying advanced information technology to financial business, enables bank business personnel to obtain a large amount of important information related to customers in real time, thereby facilitating operation and improving efficiency.

There are still defects in the requirements of incremental data and real-time data query in the data warehouse.

There is no effective solution to the problem of poor performance of Internet financial data storage and calculation processing in related technologies.

SUMMARY OF THE INVENTION

The main purpose of this application is to provide a data processing method and device, a storage medium and an electronic device for a data warehouse, so as to solve the problem of poor performance of Internet financial data storage and calculation processing, and no effective problem has yet been proposed.

In order to achieve the above object, according to an aspect of the present application, a data processing method for a data warehouse is provided.

The data processing method for a data warehouse according to the present application includes: acquiring data from different data sources based on the Flink framework, and storing it according to the data lake hudi; establishing a target data warehouse based on preset data extraction, transformation and loading processing ; Based on the target data warehouse, perform a preset data processing operation, wherein the preset data processing operation includes at least one of the following: an operation of generating a report, an operation of querying business data, and an operation of cleaning raw data of a decision engine.

Further, establishing the target data warehouse based on the preset data extraction, conversion and loading processing also includes: establishing the target data warehouse based on the preset full data and/or incremental data extraction, conversion and loading processing, wherein , the full/incremental data is extracted to the kafka queue using the Canel component.

Further, establishing the target data warehouse based on the preset data extraction, conversion and loading process includes: loading the data into the target data warehouse by reading data from the data source, performing data type conversion and cleaning dirty data.

Further, performing a preset data processing operation based on the target data warehouse further includes: a user behavior data analysis operation, printing the business data to a fixed file through a preset data model, and collecting the log file; Extract the log file to the kafka queue; pass the data into the target data warehouse through the input source of the target data warehouse, and use Flink SQL to clean the data, and then organize the data into the required data and store it in Hbase.

Further, the preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: a report generation operation, a business data query operation, and a decision engine raw data cleaning operation, Including: the preset data processing operation includes: generating a report operation, after accessing the input source through the target data warehouse, cleaning the data based on SQL to obtain data required by the target format; accessing through the target data warehouse Output the source, store the data, and display the data through BI tools.

Further, the preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: a report generation operation, a business data query operation, and a decision engine raw data cleaning operation, Including: the preset data processing operation includes: a decision engine raw data cleaning operation, accessing a data source through the target data warehouse, and using a preset Flink SQL to clean the data, wherein the Flink SQL is configured according to business data, The result of each Flink SQL execution is stored in the data lake hudi as intermediate data; the data is cleaned into the original field of the preset decision, and stored in the target data warehouse through the output source; for incremental data, the update field is used to extract and transform and a loading process to establish the target data warehouse.

Further, the preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: a report generation operation, a business data query operation, and a decision engine raw data cleaning operation, It includes: the preset data processing operation includes: a business data query operation, by extracting the full amount of data to the kafka queue; cleaning the data from the data warehouse through the input source; and configuring the ES as the output source for the cleaning result.

In order to achieve the above object, according to another aspect of the present application, a data processing device for a data warehouse is provided.

The data processing device for data warehouses according to the present application includes: an establishment module for establishing a target data warehouse based on preset data extraction, conversion and loading processing; an execution processing module for executing the execution based on the target data warehouse A preset data processing operation, wherein the preset data processing operation includes at least one of the following: a report generation operation, a business data query operation, and a decision engine raw data cleaning operation.

In order to achieve the above object, according to another aspect of the present application, a computer-readable storage medium is provided, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the method.

In order to achieve the above object, according to yet another aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the computer program. method described.

In the data processing method, device, storage medium, and electronic device used in the data warehouse in the embodiments of this application, data is obtained from different data sources based on the Flink framework, and stored according to the data lake hudi; based on preset data extraction , conversion and loading processing, and the method of establishing a target data warehouse, by executing a preset data processing operation based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: report generation operation, business data query Operation and decision-making engine raw data cleaning operation achieves the purpose of completing report processing, timely query of business data, and cleaning of decision-making engine raw data, thus realizing the technical effect of optimizing data storage and calculation process, and solving the problem of Internet financial data storage. As well as technical issues with poor computational processing.

Description of drawings

The accompanying drawings, which constitute a part of this application, are used to provide a further understanding of the application and make other features, objects and advantages of the application more apparent. The accompanying drawings and descriptions of the exemplary embodiments of the present application are used to explain the present application, and do not constitute an improper limitation of the present application. In the attached image:

1 is a schematic diagram of a hardware structure of a data processing method for a data warehouse according to an embodiment of the present application;

2 is a schematic flowchart of a data processing method for a data warehouse according to an embodiment of the present application;

3 is a schematic structural diagram of a data processing apparatus for a data warehouse according to an embodiment of the present application;

FIG. 4 is a schematic flowchart of a data processing method for a data warehouse according to an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

In this application, the terms "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "middle", The orientation or positional relationship indicated by "vertical", "horizontal", "horizontal", "longitudinal", etc. is based on the orientation or positional relationship shown in the drawings. These terms are primarily used to better describe the present application and its embodiments, and are not intended to limit the fact that the indicated device, element, or component must have a particular orientation, or be constructed and operated in a particular orientation.

In addition, some of the above-mentioned terms may be used to express other meanings besides orientation or positional relationship. For example, the term "on" may also be used to express a certain attachment or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in the present application can be understood according to specific situations.

Furthermore, the terms "installed", "set up", "provided with", "connected", "connected", "socketed" should be construed broadly. For example, it may be a fixed connection, a detachable connection, or a unitary structure; it may be a mechanical connection, or an electrical connection; it may be directly connected, or indirectly connected through an intermediary, or between two devices, elements, or components. internal communication. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific situations.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

As shown in FIG. 1, it is a schematic diagram of a hardware structure of a data processing method for a data warehouse according to an embodiment of the present application, including that the data warehouse 100 is a theme-oriented, integrated and relatively stable data storage collection reflecting historical changes, used for Support the analysis report and decision-making of the enterprise. The data in the data warehouse comes from the integration of different data sources, such as data source 1, data source 2, and data source 3, and the storage methods of these data sources may be different, such as mysql, orcale, hive, etc., so Etl needs to be done Operations integrate disparate data sources. The Etl operation includes, but is not limited to, data extraction, which is to read data from the data source. Data conversion, data type conversion and dirty data cleaning. Data loading, the processed data is loaded into the target, such as a data warehouse. Based on the data warehouse 100, business functions such as data reporting, data mining, and data analysis can be provided.

As shown in Figure 2, the method includes the following steps S201 to S202:

Step S201, obtain data from different data sources based on the Flink framework, and store it according to the data lake hudi;

Step S202, establishing a target data warehouse based on preset data extraction, conversion and loading processing;

Step S203: Execute a preset data processing operation based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: an operation of generating a report, an operation of querying business data, and an operation of cleaning raw data of a decision engine.

From the above description, it can be seen that the application has achieved the following technical effects:

By adopting the method of establishing a target data warehouse based on preset data extraction, conversion and loading processing, a preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: The report generation operation, business data query operation, and decision engine raw data cleaning operation achieve the purpose of completing report processing, timely query of business data, and cleaning the original data of decision engine, thereby realizing the technical effect of optimizing data storage and calculation process, and then It solves the technical problems of the storage of Internet financial data and the poor calculation and processing effect.

After the input source is configured to the data warehouse (platform) in the above steps S201 and S202, the data warehouse (platform) will extract the data according to the configured data source, and put the extracted data into the database lake hudi for cleaning, Then output to the configured data source.

As a preference in this embodiment, the establishment of a target data warehouse based on preset data extraction, conversion and loading processing includes: extracting data from data sources in a preset component-based integration manner, and storing data based on data lake hudi. The method is converted and loaded based on the Flink technical framework; by reading the data from the data source, converting the data type and cleaning the dirty data, it is loaded into the target data warehouse.

As a preferred implementation, the Flink technology framework is used to carry the entire data warehouse system, the data lake hudi is used for data storage, the data source is integrated by components, and conventional data source operations are supported. Data sources include but are not limited to oracle, mysql, hbase, kafka, es, mongodb, redis.

As a preferred embodiment, report processing is completed based on the data warehouse, business data is queried in time, and raw data of the decision engine is cleaned.

As a preferred implementation, based on the lightweight and easy migration deployment of the data warehouse.

As a preferred implementation manner, the visualization configuration is simple under different preset business scenarios based on the data warehouse. The preset business scenarios include, but are not limited to, real-time data requirements, data tracking and other business scenarios. That is, it supports visual configuration.

In the above step S202, a preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: an operation of generating a report, an operation of querying business data, and an operation of cleaning raw data of a decision engine.

As a preferred implementation, on the incremental data processing mechanism, the CDC mechanism of the database is used to extract data, and some databases without the CDC mechanism use the Kafka queue to synchronize the data to the Kafka queue first, and then store the data in the data warehouse (platform). Configure the kafka queue as an input source for data cleaning operations.

As a preferred implementation, for business data query operations, the buried point log is printed to a fixed file in the business system, synchronized to kafka through a file extraction tool, and then the kafka input source is configured on the data warehouse platform to clean the data.

As a preferred implementation, for the raw data cleaning operation of the decision engine, whether database data or kafka data is used as the data input source, and then Etl cleaning is performed through Flink SQL. This SQL can be configured for each business, and each SQL executes The results are stored in the data lake as intermediate data.

As a preference in this embodiment, establishing the target data warehouse based on the preset data extraction, transformation and loading processing further includes: based on the preset full data and/or incremental data extraction, transformation and loading processing, establishing The target data warehouse, in which the full/incremental data is extracted to the kafka queue using the Canel component.

During specific implementation, considering that there are full data and incremental data in different scenarios, it is necessary to consider synchronizing the two parts of data when establishing a data warehouse. That is, full/incremental data is extracted to the kafka queue using the Canel component. The full and incremental data here are extracted to kafka using the Canel component.

As a preference in this embodiment, the performing a preset data processing operation based on the target data warehouse further includes: a user behavior data analysis operation, printing the business data and the buried point log to a fixed file by using a preset data model, Collect the log files; extract the log files to the Kafka queue; pass the data into the target data warehouse through the input source of the target data warehouse, and use Flink SQL to clean the data, and then organize the data into the required data Data is stored in Hbase.

In the specific implementation, first collect business data into log files through the data model, then use the = component to extract the logs to kafka, and then use the input source connected to the data warehouse platform to transfer the data to the data warehouse platform and use Flink SQL to clean the data. Then organize the data into the required data and put it into Hbase for business use.

As a preference in this embodiment, the preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: report generation operation, business data query operation, decision engine operation The original data cleaning operation includes: the preset data processing operation includes: generating a report operation, after accessing the input source through the target data warehouse, cleaning the data based on SQL to obtain the data required by the target format; The target data warehouse is connected to the output source, stores the data, and displays the data through BI tools.

In the specific implementation, the input source is accessed through the data warehouse platform, and the data is cleaned through SQL, which may generate intermediate tables. After multiple cleanings, various data required by the target format are finally formed. Data is stored and displayed through BI tools.

As a preference in this embodiment, the preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: report generation operation, business data query operation, decision engine operation The original data cleaning operation includes: the preset data processing operation includes: the decision engine original data cleaning operation, accessing the data source through the target data warehouse, and using the preset Flink SQL to clean the data, wherein the Flink SQL according to The business data is configured, and the result of each Flink SQL execution is stored in the data lake hudi as intermediate data; the data is cleaned into the original field of the preset decision, and stored in the target data warehouse through the output source; for incremental data, use update Fields are extracted, converted and loaded to establish the target data warehouse.

In the specific implementation, the data source is accessed through the data warehouse platform, and the data is cleaned by SQL, and the data is cleaned into the original fields for decision-making, and stored in the target database through the output source. For incremental data, use update fields to extract to the data warehouse (platform).

As a preference in this embodiment, the preset data processing operation is performed based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: report generation operation, business data query operation, decision engine operation The original data cleaning operation includes: the preset data processing operation includes: business data query operation, by extracting the full amount of data to the kafka queue; cleaning the data from the data warehouse through the input source; the cleaning result is configured as an output by ES source.

In specific implementation, business data has high requirements for real-time data, and it is necessary to extract the full amount of data to the Kafka queue. The data warehouse (platform) cleans the data through the input source. After cleaning, configure ES as the output source to facilitate business operations. retrieval of data.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

According to an embodiment of the present application, a data processing apparatus for data warehouses for implementing the above method is also provided. As shown in FIG. 3 , the apparatus includes:

The data processing module 301 is used to obtain data from different data sources based on the Flink framework, and store it according to the data lake hudi;

The establishment module 302 is used for establishing the target data warehouse based on the preset data extraction, conversion and loading processing;

The execution processing module 303 is configured to execute a preset data processing operation based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: report generation operation, business data query operation, decision engine raw data cleaning operate.

After the input source is configured to the data warehouse (platform) in the data processing module 301 and the establishment module 302 of this application, the data warehouse (platform) will extract the data according to the configured data source, and extract the extracted data. The data is put into the database lake hudi for cleaning, and then output to the configured data source.

As a preference in this embodiment, the establishment of a target data warehouse based on preset data extraction, conversion and loading processing includes: extracting data from data sources in a preset component-based integration manner, and storing data based on data lake hudi. The method is converted and loaded based on the Flink technical framework; by reading the data from the data source, converting the data type and cleaning the dirty data, it is loaded into the target data warehouse.

As a preferred implementation, the Flink technology framework is used to carry the entire data warehouse system, the data lake hudi is used for data storage, the data source is integrated by components, and conventional data source operations are supported. Data sources include but are not limited to oracle, mysql, hbase, kafka, es, mongodb, redis.

As a preferred embodiment, report processing is completed based on the data warehouse, business data is queried in time, and raw data of the decision engine is cleaned.

As a preferred implementation, based on the lightweight and easy migration deployment of the data warehouse.

As a preferred implementation manner, the visualization configuration is simple under different preset business scenarios based on the data warehouse. The preset business scenarios include, but are not limited to, real-time data requirements, data tracking and other business scenarios.

The execution processing module 302 of the present application executes a preset data processing operation based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: an operation of generating a report, an operation of querying business data, and a decision-making operation. Engine raw data cleaning operations.

As a preferred implementation, on the incremental data processing mechanism, the CDC mechanism of the database is used to extract data, and some databases without the CDC mechanism use the Kafka queue to synchronize the data to the Kafka queue first, and then store the data in the data warehouse (platform). Configure the kafka queue as an input source for data cleaning operations.

As a preferred implementation, for business data query operations, the buried point log is printed to a fixed file in the business system, synchronized to kafka through a file extraction tool, and then the kafka input source is configured on the data warehouse platform to clean the data.

As a preferred implementation, for the raw data cleaning operation of the decision engine, whether database data or kafka data is used as the data input source, and then Etl cleaning is performed through Flink SQL. This SQL can be configured for each business, and each SQL executes The results are stored in the data lake as intermediate data.

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general-purpose computing device, and they can be centralized on a single computing device, or distributed in a network composed of multiple computing devices Alternatively, they can be implemented with program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or they can be integrated into The multiple modules or steps are fabricated into a single integrated circuit module. As such, the present application is not limited to any particular combination of hardware and software.

In order to better understand the flow of the above-mentioned intelligent dialogue interaction method, the above-mentioned technical solutions are explained below with reference to the preferred embodiments, but are not used to limit the technical solutions of the embodiments of the present invention.

The data processing method for a data warehouse in the embodiment of the present application completes report processing based on the data warehouse (platform), timely query of business data, and cleaning of raw data of a decision engine. It is based on lightweight and easy to migrate and deploy, and the visual configuration is simple in different business scenarios.

As shown in FIG. 4 , it is a schematic flowchart of a data processing method for a data warehouse in an embodiment of the present application. The specific process of implementation includes the following steps:

In step S401, a target data warehouse is established based on preset data extraction, conversion and loading processing.

Step S402, whether there is incremental data.

Step S403, based on the preset full data and/or incremental data extraction, conversion and loading processing, establish a target data warehouse, wherein the full/incremental data is extracted to the kafka queue using the Canel component.

In the incremental data processing mechanism, the CDC mechanism of the database is used to extract data. Some databases without the CDC mechanism use Kafka to synchronize the data to Kafka first, and then configure Kafka as an input source on the data warehouse platform for data cleaning operations.

For process tracking business data, print the buried log to a fixed file in the business system, synchronize it to kafka through the file extraction tool, and then configure the kafka queue input source on the data warehouse platform to clean the data.

The specific data cleaning process, whether it is database data or kafka queue data as the data input source, and then Etl cleaning through Flink SQL, can be configured for each business based on SQL, and the result of each SQL execution is stored in the data lake as intermediate data Step S404, perform data extraction from the data source according to the preset component-based integration, conversion based on the data lake hudi storage, and loading processing based on the Flink technical framework.

In step S405, the data is loaded into the target data warehouse by reading out the data from the data source, converting the data type and cleaning the dirty data.

Step S406: Execute a preset data processing operation based on the target data warehouse, wherein the preset data processing operation includes at least one of the following: an operation of generating a report, an operation of querying business data, and an operation of cleaning raw data of a decision engine.

Generate report operations, access the input source through the data warehouse platform, and clean the data through SQL, which may generate intermediate tables. After multiple cleanings, various data required by the target format are finally formed. Data is stored and displayed through BI tools.

The original data cleaning operation of the decision engine, access the data source through the data warehouse platform, use SQL to clean the data, clean the data into the original fields of the decision, and store it in the target database through the output source. For incremental data, use the update field to extract to the data warehouse platform.

For business data query operations, business data has high real-time requirements for data. It is necessary to extract full data to Kafka, and the data warehouse (platform) cleans the data through the input source. After cleaning, configure ES as the output source, which is convenient for business operations. retrieval of data. The full and incremental data here is extracted to the kafka queue using the Canel component.

In addition, for user behavior data analysis operations, first collect business data into log files through the data model, then the component extracts the logs to the Kafka queue, and then uses the input source to the data warehouse platform to transfer the data to the data warehouse (platform) using Flink SQL cleans the data, and then organizes the data into the required data and puts it into Hbase for business use.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (10)

1. A data processing method for a data bin, comprising:

acquiring data from different data sources based on a Flink frame, and storing the data according to the data lake hudi;

establishing a target data bin based on preset data extraction, conversion and loading processing;

executing preset data processing operation based on the target data bin, wherein the preset data processing operation at least comprises one of the following operations: generating report operation, business data query operation and decision engine original data cleaning operation.

2. The method of claim 1, wherein establishing a target data bin based on a pre-defined data extraction, conversion, and loading process further comprises:

and establishing a target data bin based on preset full-amount data and/or incremental data extraction, conversion and loading processes, wherein the full-amount/incremental data is extracted to the kafka queue by using a Canel component.

3. The method of claim 2, wherein establishing a target data bin based on a pre-defined data extraction, conversion and loading process comprises:

the data is loaded to a target data bin after being read out from a data source and subjected to data type conversion and dirty data cleaning.

4. The method of claim 1, wherein performing a pre-set data processing operation based on the target data bin further comprises: the user behavior data analysis operation is performed,

printing the service data to a fixed file by a preset data model, and collecting a log file;

extracting the log file to a kafka queue;

and transmitting the data into the target data bin through the input source of the target data bin, cleaning the data by adopting the Flink SQL, then sorting the data into the required data and storing the data into the Hbase.

5. The method of claim 1, wherein the performing of the pre-set data processing operation is based on the target data bin, wherein the pre-set data processing operation comprises at least one of: generating report operation, business data query operation and decision engine original data cleaning operation, comprising:

the preset data processing operation comprises: the operation of generating a report is carried out,

after an input source is accessed through the target data bin, cleaning data based on SQL to obtain data required by a target format;

and accessing an output source through the target data bin, storing the data, and displaying the data through a BI tool.

6. The method of claim 1, wherein the performing of the pre-set data processing operation is based on the target data bin, wherein the pre-set data processing operation comprises at least one of: generating report operation, business data query operation and decision engine original data cleaning operation, comprising:

the preset data processing operation comprises: the decision engine raw data is flushed of operations,

accessing a data source through the target data bin, and cleaning data by using preset Flink SQL, wherein the Flink SQL is configured according to service data, and the execution result of each Flink SQL is stored into the data lake hudi as intermediate data;

cleaning data into a preset decision original field, and storing the data into the target data bin through an output source;

and performing extraction, conversion and loading processing on incremental data by using the updated field, and establishing the target data bin.

7. The method of claim 1, wherein the performing of the pre-set data processing operation is based on the target data bin, wherein the pre-set data processing operation comprises at least one of: generating report operation, business data query operation and decision engine original data cleaning operation, comprising:

the preset data processing operation comprises: a business data query operation is performed on the business data,

by extracting the full amount of data to the kafka queue;

cleaning the data bin through an input source;

the cleaning result is used as an output source by configuring the ES.

8. A data processing apparatus for counting bins, comprising:

the data processing module is used for acquiring data from different data sources based on a Flink frame and storing the data according to the data lake hudi;

the establishing module is used for establishing a target data bin based on preset data extraction, conversion and loading processing;

an execution processing module, configured to execute a preset data processing operation based on the target data bin, where the preset data processing operation at least includes one of: generating report operation, business data query operation and decision engine original data cleaning operation.

9. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 7.

Images