CN113778999A - Data modularization processing method and device, computer equipment and storage medium - Google Patents
Data modularization processing method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113778999A CN113778999A CN202111151829.8A CN202111151829A CN113778999A CN 113778999 A CN113778999 A CN 113778999A CN 202111151829 A CN202111151829 A CN 202111151829A CN 113778999 A CN113778999 A CN 113778999A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- result
- type
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 288
- 238000000034 method Methods 0.000 claims abstract description 43
- 230000001419 dependent effect Effects 0.000 claims abstract description 40
- 238000013075 data extraction Methods 0.000 claims abstract description 8
- 238000004590 computer program Methods 0.000 claims description 26
- 238000001914 filtration Methods 0.000 claims description 23
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 abstract description 22
- 238000011161 development Methods 0.000 abstract description 5
- 238000004364 calculation method Methods 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 210000001520 comb Anatomy 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to the field of development-assisted processing templates, and in particular, to a data modular processing method, apparatus, computer device, and storage medium. After a data processing request is obtained, determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing the type of the required data; determining dependent data required by the result data in the target data warehouse according to the data template; acquiring data to be converted corresponding to the result data according to the dependent data; and performing modular processing on the data to be converted according to the data of the required data type to obtain result data. According to the data processing method and device, the dependence data required by the result data are determined firstly, then the data are processed and extracted, and the data are directly extracted through the comparison script, so that repeated calculation in the data extraction process is avoided, and the data acquisition efficiency from the data warehouse is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data modularization processing method and apparatus, a computer device, and a storage medium.
Background
With the development of computer technology, big data technology is also continuously developed. Big data (big data), or huge data, refers to the data that is too large to be captured, managed, processed and organized in a reasonable time to help the enterprise to make business decisions more positive by the current mainstream software tools. In the big data era, data plays an important role as raw materials and is a precondition for developing a lot of businesses. While data processing for big data can be divided into two phases: the method comprises a data warehouse stage, wherein data processing in the data warehouse stage mainly comprises the steps of extracting, cleaning, converting and loading scattered, messy and non-uniform data into a data warehouse; the second stage is data processing in the process of model development and debugging, and common operations such as missing value processing, category variable processing and the like are performed.
The data transfer process from the first phase to the second phase is generally not appreciated. Currently, it is common practice to directly script to extract the required data from the database one by one. However, the data processing method neglects the dependency relationship among the data, easily causes repeated processing of the data, consumes a long time, and increases the overall time consumption of the model.
Disclosure of Invention
In view of the above, it is necessary to provide a data modular processing method, an apparatus, a computer device and a storage medium capable of efficiently extracting data from a data warehouse.
A method of modular processing of database data, the method comprising:
acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing a required data type;
determining dependent data required by the result data in the target data warehouse according to the data template;
acquiring data to be converted corresponding to result data according to the dependent data;
and performing modular processing on the data to be converted according to the type of the required data to obtain result data.
In one embodiment, before the obtaining the data processing request and determining the target data warehouse and the data template corresponding to the data processing request, the method further includes:
and acquiring template element information, and constructing a data template according to the template element information.
In one embodiment, the performing modular processing on the data to be converted according to the type of the required data, and acquiring result data includes:
performing serial processing on the data to be converted to obtain a serial processing result;
and performing type-based modular processing on the serial processing result according to the type of the required data to obtain result data.
In one embodiment, the performing serial processing on the data to be converted to obtain a serial processing result includes:
filtering the data to be converted to obtain a filtering result;
and acquiring a source data table corresponding to the data to be converted, and performing multi-table combined access on the filtering processing result based on the source data table corresponding to the data to be converted to acquire a serial processing result.
In one embodiment, the performing type-based modular processing on the serial processing result according to the type of the required data, and acquiring result data includes:
determining a processing module corresponding to the serial processing result according to the type of the required data;
and processing the serial processing result through the processing module to obtain result data.
In one embodiment, the processing module includes at least one of a base module, an operation module, a comparison module, a trend module, and a ranking module.
In one embodiment, the performing type-based modular processing on the serial processing result according to the type of the required data, and acquiring result data includes:
determining each processing module corresponding to the serial processing result and the processing sequence of each processing module according to the type of the required data;
and processing the serial processing result sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data.
A data modular processing apparatus, the apparatus comprising:
the data processing system comprises a request acquisition unit, a data processing unit and a data processing unit, wherein the request acquisition unit is used for acquiring a data processing request and determining a target data warehouse and a data template corresponding to the data processing request, and the data template carries data used for representing a required data type;
the dependent data searching unit is used for determining dependent data required by the result data in the target data warehouse according to the data template;
the data extraction unit is used for acquiring data to be converted corresponding to the result data according to the dependent data;
and the data conversion unit is used for performing modular processing on the data to be converted according to the type of the required data to obtain result data.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing a required data type;
determining dependent data required by the result data in the target data warehouse according to the data template;
acquiring data to be converted corresponding to result data according to the dependent data;
and performing modular processing on the data to be converted according to the type of the required data to obtain result data.
A computer storage medium having a computer program stored thereon, the computer program when executed by a processor implementing the steps of:
acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing a required data type;
determining dependent data required by the result data in the target data warehouse according to the data template;
acquiring data to be converted corresponding to result data according to the dependent data;
and performing modular processing on the data to be converted according to the type of the required data to obtain result data.
The data modularization processing method, the data modularization processing device, the computer equipment and the storage medium are characterized in that after a data processing request is obtained, a target data warehouse and a data template corresponding to the data processing request are determined, wherein the data template carries data used for representing a required data type; determining dependent data required by the result data in the target data warehouse according to the data template; acquiring data to be converted corresponding to the result data according to the dependent data; and performing modular processing on the data to be converted according to the data of the required data type to obtain result data. The method and the device determine the dependency data from the target data warehouse by configuring the data template, copy the data to be converted, process the data to be converted according to the type of the required data, generate the result data in a modularized mode, determine the dependency data required by the result data firstly, process and extract the data, and directly extract the data by the comparison script, so that repeated calculation in the data extraction process is avoided, and the efficiency of obtaining the data from the data warehouse is improved.
Drawings
FIG. 1 is a diagram illustrating an exemplary implementation of a modular data processing method;
FIG. 2 is a flow diagram illustrating a data modular processing method according to one embodiment;
FIG. 3 is a schematic sub-flow chart of step 207 of FIG. 2 in one embodiment;
FIG. 4 is a schematic sub-flow chart of step 302 of FIG. 3 in one embodiment;
FIG. 5 is a schematic sub-flow chart of step 304 of FIG. 3 in one embodiment;
FIG. 6 is a block diagram of a data modular processing apparatus in one embodiment;
FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result. The artificial intelligence technology in the application is mainly used for further processing and applying result data obtained by modular processing.
The application specifically provides a data modularization processing method, which can be applied to an application environment as shown in fig. 1. Wherein, the terminal 102 can communicate with the data modular processing server 104 through the network, and the terminal 102 can send a data processing request to the data modular processing server 104. The data modular processing server 104 receives the data processing request, determines a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing the type of the required data; determining dependent data required by the result data in the target data warehouse according to the data template; acquiring data to be converted corresponding to the result data according to the dependent data; and performing modular processing on the data to be converted according to the data of the required data type to obtain result data. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the data modular processing server 104 may be an independent server, or a cloud server that provides basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, Content Distribution Network (CDN), and a big data and artificial intelligence platform.
In one embodiment, as shown in fig. 2, a data modularization processing method is provided, which is described by taking the method as an example applied to the data modularization processing server 104 in fig. 1, and includes the following steps:
step 201, acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, where the data template carries data for representing a required data type.
The data processing request is used for requesting the data processing server to process the specified data, and the target data warehouse is the position of the original data processed by the data processing request. A requestor of a data processing request may designate a data warehouse as a target data warehouse. The data template is mainly used for extracting data from the data warehouse, and the data required by subsequent big data processing can be effectively sorted out from the target data warehouse at a high speed through the data template. The required data types are data required by the templating process, which are divided into different types according to the requirements of the templating process of the data in advance, and the required data types include but are not limited to: base type, operation type, comparison type, trend type, and sort type 1) base type: the type data is the existing data in the database and can be directly obtained from the database; 2) operation type: the type data does not directly exist in the database and needs to be obtained by utilizing relevant operations, such as addition, subtraction, multiplication, division and the like; 3) the comparison type is as follows: the type represents the change condition of the data at two time points needing to be compared and does not directly exist in the database; 4) the trend types are: the type data represents the change of a certain parameter in the past period, does not directly exist in a database, and needs to be written with codes for implementation; 5) the sort type is as follows: the type data represents the sequence of a certain individual in a group, does not directly exist in a database, and needs to be written with codes for implementation; 6) type XX: and the user defines the data type. The required data type may be any one of the above types, or multiple types, and when the required data type includes multiple types, the data template further needs to correspondingly specify an order corresponding to the types, so as to determine a processing order of the corresponding processing module.
Specifically, the data modularization processing of the present application is applicable to large data level processing of data in a data warehouse. Currently, data processing for big data can be divided into two phases: the method comprises a data warehouse stage, wherein data processing in the data warehouse stage mainly comprises the steps of extracting, cleaning, converting and loading scattered, messy and non-uniform data into a data warehouse; the second stage is data processing in the process of model development and debugging, and common operations such as missing value processing, category variable processing and the like are performed. The data processing method is specifically used for converting the data processed in the first stage into modular data for the data processing in the second stage. Therefore, after the data processing request is obtained, the target data warehouse and the data template corresponding to the data processing request need to be determined, then the data modularization processing is performed on the specified data in the target data warehouse based on the data template, and the processed data is used as the basic data for data processing in the subsequent process of model development and debugging.
And step 203, determining the required dependent data of the result data in the target data warehouse according to the data template.
Wherein, the dependent data refers to the existing data in the data table in the target data warehouse. The data required by the data template can be obtained by processing the dependent data to a certain extent.
Specifically, the main work of extracting data according to the data template is to take out dependent data corresponding to the data in the data template from a data table of the data warehouse, and then obtain templated data required by subsequent big data calculation through certain processing, so that after the data template is obtained, the dependent data corresponding to each type of data required by the data template can be determined according to the data name and the source data table in the data template.
And step 205, acquiring data to be converted corresponding to the result data according to the dependent data.
The data to be converted refers to the data which is copied from the target data warehouse data table and has the same dependent data. Acquiring the data to be converted corresponding to the result data according to the dependent data specifically means copying the dependent data in the target data warehouse. Specifically, a blank data table may be established in advance, and after determining the dependent data, the dependent data in the original database may be copied to the blank data table in a copy manner, so as to obtain the data to be converted.
Specifically, step 205 essentially combs the data in the configuration template, integrates all dependent data in the target data warehouse in units of database tables, and reads the data. The union of the constraints of the data contained in the same database table serves as the constraint of the database table. And taking the table in the target data warehouse as a unit, and extracting data from the target data warehouse to obtain the data to be converted.
And step 207, performing modular processing on the data to be converted according to the data of the required data type to obtain result data.
Specifically, after the data to be converted is extracted from the database table of the target data warehouse, the data to be converted may be subjected to modular processing by data processing based on the required data type corresponding to each type of data in the data template, so as to obtain result data. The conversion into result data through the modular processing specifically means that the data to be converted is converted through each preset data processing module according to the type of the data specified in the data template, for example, the obtained basic type data is input into the basic type module, and the obtained operation type data is input into the operation type module.
The data modular processing method comprises the steps of determining a target data warehouse and a data template corresponding to a data processing request after the data processing request is obtained, wherein the data template carries data for representing the type of the required data; determining dependent data required by the result data in the target data warehouse according to the data template; acquiring data to be converted corresponding to the result data according to the dependent data; and performing modular processing on the data to be converted according to the data of the required data type to obtain result data. The method and the device determine the dependency data from the target data warehouse by configuring the data template, copy the data to be converted, process the data to be converted according to the type of the required data, generate the result data in a modularized mode, determine the dependency data required by the result data firstly, process and extract the data, and directly extract the data by the comparison script, so that repeated calculation in the data extraction process is avoided, and the efficiency of obtaining the data from the data warehouse is improved.
In one embodiment, before step 201, the method further includes: and acquiring template element information, and constructing a data template according to the template element information.
The template element information is the main constituent content for the data template, and includes data name, source table, generation logic, constraint condition, required data type, and the like. The data name refers to a name used for data in the service. The source table refers to the table name of the data in the database, and if the data depends on one or more data, the corresponding table name is one or more. Constraints refer to limitations on the range of data, including but not limited to time spans, ranges of values for numerical variables, ranges defined by other data, and the like.
Specifically, before data extraction and data modularization processing are performed based on the data template, a corresponding data template needs to be constructed. In this case, the data template may be specifically constructed based on template element information, and the data template may be constructed by specifying information such as a data name, a source table, generation logic, a constraint condition, and a type to which the data belongs. Specifically, a preset blank data template may be acquired first, and then corresponding data in the blank data template may be set in the future according to the acquired template element information, so as to construct an available data template. In this embodiment, the data template is constructed by the template element information, so that the processing effectiveness of the data extraction and the modularization processing process can be more effectively ensured.
In one embodiment, as shown in FIG. 3, step 207 comprises:
and 302, performing serial processing on the data to be converted to obtain a serial processing result.
And 304, performing type-based modular processing on the serial processing result according to the data of the required data type to obtain result data.
Specifically, the processing of the data to be converted may specifically include two steps of serial processing and type-based modular processing. The serial processing is a unified processing of all data to be converted, and includes operations that data filtering and the like can be performed in series and in steps. The classification type modularization processing is to extract corresponding data to be converted according to the type of the data, and then perform corresponding modularization processing through different data processing modules, and in the process, different data to be converted are required to be input into different data processing modules for processing, so that various types of result data corresponding to the type of the data are obtained. In the embodiment, the result data is processed in a segmented manner through serial processing and classified modular processing, so that various types of result data can be obtained more effectively.
In one embodiment, as shown in FIG. 4, step 302 comprises:
step 401, filtering the data to be converted to obtain a filtering result.
And 403, acquiring a source data table corresponding to the data to be converted, and performing multi-table combined access on the filtering processing result based on the source data table corresponding to the data to be converted to acquire a serial processing result.
The filtering process is the adjustment of the range of the data itself during data fetching, and specifically means that data outside the requirement is filtered from the data to be converted according to a preset filtering requirement. When the filtering processing is carried out, each item of data to be converted of a single table comprises a preset requirement numerical value, then the data to be converted are filtered based on the preset requirement numerical values, and the data to be converted which do not meet the requirement are deleted. The processing of multi-table join access refers to adjustment of influence on final data based on two or more tables join access, and at this time, the access needs to be integrated by combining preset demand values corresponding to each table in the multi-table join. In the embodiment, through data filtering and multi-table combined processing, useless data in the data to be converted can be effectively removed, so that the usability of finally obtained result data is ensured.
In one embodiment, as shown in FIG. 5, step 304 includes:
step 502, determining a processing module corresponding to the serial processing result according to the data of the required data type.
Step 504, the serial processing result is processed by the processing module to obtain result data.
Specifically, the performing type-based modular processing on the serial processing result specifically includes: when various types of result data are obtained, the generation process of each type of result data can be processed through an independent processing module. The data type of the result data is specifically consistent with the type of the data specified by the data template, and in a specific embodiment, the processing module specifically includes a basic module, an operation module, a comparison module, a trend module, and a sorting module. The processing of the serial processing result by the processing module specifically means that for the basic type data, the basic module can be selected for processing and obtaining; acquiring operation type data by adopting an operation module; for the comparison type data, a comparison module is adopted for obtaining; acquiring trend type data by adopting a trend module; and for the sequencing type data, a sequencing type data generation module is adopted for obtaining. In addition, the user can design some self-defined modules to perform some more complicated processing, and at the moment, the data can be processed through the user-defined modules. In the processing, the processing of the serial processing result by the processing module specifically refers to the processing of the serial processing result by any one of the modules. In this embodiment, various types of result data can be effectively obtained by processing the data to be converted through various types of processing modules, and the availability of the result data is ensured.
In one embodiment, step 304 includes: determining each processing module corresponding to the serial processing result and the processing sequence of each processing module according to the data of the required data type; and processing the serial processing results sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data.
Specifically, the serial processing result is processed in a type-specific modular manner, and besides being processed by a single processing module, the serial processing result may be combined by two or more modules, and the required modules are called in a certain order. At this time, the processing modules corresponding to the serial processing result and the processing sequence of each processing module need to be determined according to the type of the data template, and the processing sequence of each processing module can be preset in the data template according to actual requirements; and processing the serial processing results sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data. For example, in a specific embodiment, when a certain type of data is calculated, the comparison type module may be called first, and then the trend module may be called, and the result data is finally obtained. In this embodiment, by collecting a plurality of processing modules and setting the processing order of the processing modules to process the data to be converted, various types of result data can be effectively obtained, and the availability of the result data is ensured.
It should be understood that although the various steps in the flow charts of fig. 2-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 6, there is provided a data modular processing apparatus including:
the request obtaining unit 601 is configured to obtain a data processing request, and determine a target data warehouse and a data template corresponding to the data processing request, where the data template carries data used for representing a type of required data.
And a dependent data searching unit 603, configured to determine, according to the data template, dependent data required by the result data in the target data warehouse.
And the data extraction unit 605 is configured to obtain data to be converted corresponding to the result data according to the dependent data.
The data conversion unit 607 is configured to perform modular processing on the data to be converted according to the data of the required data type, and obtain result data.
In one embodiment, the system further comprises a template building module, configured to: and acquiring template element information, and constructing a data template according to the template element information.
In one embodiment, the data conversion unit 607 is specifically configured to: carrying out serial processing on data to be converted to obtain a serial processing result; and performing type-based modular processing on the serial processing result according to the data of the required data type to obtain result data.
In one embodiment, the data conversion unit 607 is specifically configured to: filtering the data to be converted to obtain a filtering result; and acquiring a source data table corresponding to the data to be converted, and performing multi-table combined access on the filtering processing result based on the source data table corresponding to the data to be converted to acquire a serial processing result.
In one embodiment, the data conversion unit 607 is specifically configured to: determining a processing module corresponding to a serial processing result according to the data of the required data type; and processing the serial processing result through the processing module to obtain result data.
In one embodiment, the processing module includes a base module, an operation module, a comparison module, a trend module, and a ranking module.
In one embodiment, the data conversion unit is specifically configured to: determining each processing module corresponding to the serial processing result and the processing sequence of each processing module according to the data of the required data type; and processing the serial processing results sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data.
For specific embodiments of the data modularization processing apparatus, reference may be made to the above embodiments of the data modularization processing method, and details are not described here. The units in the data modular processing device can be wholly or partially realized by software, hardware and a combination thereof. The units can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data modularization processing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data modular processing method.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, there is provided a computer device comprising a memory storing a computer program and a processor implementing the following steps when the processor executes the computer program:
acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing a required data type;
determining dependent data required by the result data in the target data warehouse according to the data template;
acquiring data to be converted corresponding to the result data according to the dependent data;
and performing modular processing on the data to be converted according to the data of the required data type to obtain result data.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and acquiring template element information, and constructing a data template according to the template element information.
In one embodiment, the processor, when executing the computer program, further performs the steps of: carrying out serial processing on data to be converted to obtain a serial processing result; and performing type-based modular processing on the serial processing result according to the data of the required data type to obtain result data.
In one embodiment, the processor, when executing the computer program, further performs the steps of: filtering the data to be converted to obtain a filtering result; and acquiring a source data table corresponding to the data to be converted, and performing multi-table combined access on the filtering processing result based on the source data table corresponding to the data to be converted to acquire a serial processing result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: determining a processing module corresponding to a serial processing result according to the data of the required data type; and processing the serial processing result through the processing module to obtain result data.
In one embodiment, the processor, when executing the computer program, further performs the steps of: determining each processing module corresponding to the serial processing result and the processing sequence of each processing module according to the data of the required data type; and processing the serial processing results sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data.
In one embodiment, a computer storage medium is provided, having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of:
acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing a required data type;
determining dependent data required by the result data in the target data warehouse according to the data template;
acquiring data to be converted corresponding to the result data according to the dependent data;
and performing modular processing on the data to be converted according to the data of the required data type to obtain result data.
In one embodiment, the computer program when executed by the processor further performs the steps of: and acquiring template element information, and constructing a data template according to the template element information.
In one embodiment, the computer program when executed by the processor further performs the steps of: carrying out serial processing on data to be converted to obtain a serial processing result; and performing type-based modular processing on the serial processing result according to the data of the required data type to obtain result data.
In one embodiment, the computer program when executed by the processor further performs the steps of: filtering the data to be converted to obtain a filtering result; and acquiring a source data table corresponding to the data to be converted, and performing multi-table combined access on the filtering processing result based on the source data table corresponding to the data to be converted to acquire a serial processing result.
In one embodiment, the computer program when executed by the processor further performs the steps of: determining a processing module corresponding to a serial processing result according to the data of the required data type; and processing the serial processing result through the processing module to obtain result data.
In one embodiment, the computer program when executed by the processor further performs the steps of: determining each processing module corresponding to the serial processing result and the processing sequence of each processing module according to the data of the required data type; and processing the serial processing results sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A data modularization processing method is characterized by comprising the following steps:
acquiring a data processing request, and determining a target data warehouse and a data template corresponding to the data processing request, wherein the data template carries data for representing a required data type;
determining dependent data required by result data in the target data warehouse according to the data template;
acquiring data to be converted corresponding to result data according to the dependent data;
and performing modular processing on the data to be converted according to the type of the required data to obtain result data.
2. The method of claim 1, wherein before obtaining the data processing request and determining the target data warehouse and the data template corresponding to the data processing request, further comprising:
and acquiring template element information, and constructing a data template according to the template element information.
3. The method according to claim 1, wherein the modularizing the data to be transformed according to the type of the required data, and obtaining result data comprises:
performing serial processing on the data to be converted to obtain a serial processing result;
and performing type-based modular processing on the serial processing result according to the type of the required data to obtain result data.
4. The method according to claim 3, wherein the performing serial processing on the data to be converted and obtaining a serial processing result comprises:
filtering the data to be converted to obtain a filtering result;
and acquiring a source data table corresponding to the data to be converted, and performing multi-table combined access on the filtering processing result based on the source data table corresponding to the data to be converted to acquire a serial processing result.
5. The method of claim 3, wherein the performing type-specific modular processing on the serial processing result according to the type of the required data, and obtaining result data comprises:
determining a processing module corresponding to the serial processing result according to the type of the required data;
and processing the serial processing result through the processing module to obtain result data.
6. The method of claim 5, wherein the processing module comprises at least one of a base module, an arithmetic module, a comparison module, a trend module, and a ranking module.
7. The method of claim 3, wherein the performing type-specific modular processing on the serial processing result according to the type of the required data, and obtaining result data comprises:
determining each processing module corresponding to the serial processing result and the processing sequence of each processing module according to the type of the required data;
and processing the serial processing result sequentially through the processing modules according to the processing sequence of the processing modules to obtain result data.
8. A data modular processing apparatus, characterized in that the apparatus comprises:
the data processing system comprises a request acquisition unit, a data processing unit and a data processing unit, wherein the request acquisition unit is used for acquiring a data processing request and determining a target data warehouse and a data template corresponding to the data processing request, and the data template carries data used for representing a required data type;
the dependent data searching unit is used for determining dependent data required by the result data in the target data warehouse according to the data template;
the data extraction unit is used for acquiring data to be converted corresponding to the result data according to the dependency data and acquiring data to be converted corresponding to the result data according to the dependency data;
and the data conversion unit is used for performing modular processing on the data to be converted according to the type of the required data to obtain result data.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer storage medium on which a computer program is stored, characterized in that the computer program, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111151829.8A CN113778999A (en) | 2021-09-29 | 2021-09-29 | Data modularization processing method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111151829.8A CN113778999A (en) | 2021-09-29 | 2021-09-29 | Data modularization processing method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113778999A true CN113778999A (en) | 2021-12-10 |
Family
ID=78854425
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111151829.8A Pending CN113778999A (en) | 2021-09-29 | 2021-09-29 | Data modularization processing method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113778999A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107665233A (en) * | 2017-07-24 | 2018-02-06 | 上海壹账通金融科技有限公司 | Database data processing method, device, computer equipment and storage medium |
| CN110046334A (en) * | 2019-03-21 | 2019-07-23 | 平安国际智慧城市科技股份有限公司 | Data lead-in method, device, computer equipment and storage medium |
| US20200034126A1 (en) * | 2017-02-08 | 2020-01-30 | One Connect Smart Technology Co., Ltd. (Shenzhen) | File generation method, device, server, and storage medium |
| CN111241180A (en) * | 2020-01-13 | 2020-06-05 | 中国建设银行股份有限公司 | Information processing method, device, equipment and storage medium |
| CN111460773A (en) * | 2020-03-30 | 2020-07-28 | 京东数字科技控股有限公司 | Data processing method, device and system, electronic equipment and storage medium |
| CN111506559A (en) * | 2020-04-21 | 2020-08-07 | 北京同邦卓益科技有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN112732468A (en) * | 2021-04-01 | 2021-04-30 | 统信软件技术有限公司 | Data processing method, data interaction system and computing equipment |
-
2021
- 2021-09-29 CN CN202111151829.8A patent/CN113778999A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200034126A1 (en) * | 2017-02-08 | 2020-01-30 | One Connect Smart Technology Co., Ltd. (Shenzhen) | File generation method, device, server, and storage medium |
| CN107665233A (en) * | 2017-07-24 | 2018-02-06 | 上海壹账通金融科技有限公司 | Database data processing method, device, computer equipment and storage medium |
| CN110046334A (en) * | 2019-03-21 | 2019-07-23 | 平安国际智慧城市科技股份有限公司 | Data lead-in method, device, computer equipment and storage medium |
| CN111241180A (en) * | 2020-01-13 | 2020-06-05 | 中国建设银行股份有限公司 | Information processing method, device, equipment and storage medium |
| CN111460773A (en) * | 2020-03-30 | 2020-07-28 | 京东数字科技控股有限公司 | Data processing method, device and system, electronic equipment and storage medium |
| CN111506559A (en) * | 2020-04-21 | 2020-08-07 | 北京同邦卓益科技有限公司 | Data storage method and device, electronic equipment and storage medium |
| CN112732468A (en) * | 2021-04-01 | 2021-04-30 | 统信软件技术有限公司 | Data processing method, data interaction system and computing equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209652B (en) | Data table migration method, device, computer equipment and storage medium | |
| CN109032662B (en) | Code file generation method and device, computer equipment and storage medium | |
| CN110209650B (en) | Data normalization and migration method and device, computer equipment and storage medium | |
| CN111061475B (en) | Software code generating method, device, computer equipment and storage medium | |
| CN110689232A (en) | Workflow configuration optimization processing method and device and computer equipment | |
| CN103186457A (en) | Method and device for automatically generating test case | |
| JP2010524060A (en) | Data merging in distributed computing | |
| CN109389299B (en) | Workflow process deployment method, workflow process deployment device, computer equipment and storage medium | |
| CN111797604A (en) | Report generation method, device, equipment and computer readable storage medium | |
| US20180060699A1 (en) | System and method for image classification | |
| CN113220657A (en) | Data processing method and device and computer equipment | |
| CN115237857A (en) | Log processing method and device, computer equipment and storage medium | |
| CN109800965B (en) | Process business processing method and device, computer equipment and storage medium | |
| CN111950776A (en) | Business process optimization method and device, computer equipment and storage medium | |
| CN110457401B (en) | Data storage method and device, computer equipment and storage medium | |
| CN115392501A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN113535766B (en) | Workflow configuration method, device, electronic device and storage medium | |
| CN112948504B (en) | Data acquisition method and device, computer equipment and storage medium | |
| CN113590613A (en) | Data table partitioning method and device, computer equipment and storage medium | |
| CN115438768A (en) | Model reasoning method, device, computer equipment and storage medium | |
| CN113052322A (en) | Machine learning modeling method and device, storage medium and processor | |
| CN113778999A (en) | Data modularization processing method and device, computer equipment and storage medium | |
| CN115543428A (en) | Simulated data generation method and device based on strategy template | |
| CN109871214B (en) | Program code generation method, program code generation device, computer device, and storage medium | |
| CN112836033A (en) | Business model management method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20211210 |
|
| WD01 | Invention patent application deemed withdrawn after publication |