CN110795387B - Data import method, data export device and data processing system - Google Patents
Data import method, data export device and data processing system Download PDFInfo
- Publication number
- CN110795387B CN110795387B CN201910919000.4A CN201910919000A CN110795387B CN 110795387 B CN110795387 B CN 110795387B CN 201910919000 A CN201910919000 A CN 201910919000A CN 110795387 B CN110795387 B CN 110795387B
- Authority
- CN
- China
- Prior art keywords
- data
- import
- export
- processing
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data import method, which comprises the following steps: carrying out metadata configuration information on the imported target file; acquiring and analyzing metadata configuration information of an import target file to obtain import analysis configuration data; analyzing the imported target file through the adaptive data processor according to the imported analysis configuration data to obtain analysis processing file data; acquiring analysis processing file data and performing service processing on the analysis processing file data to obtain import result data; the invention also discloses a data export method, which comprises the following steps: carrying out metadata configuration information on the export target file; acquiring metadata configuration information of an export target file and analyzing the metadata configuration information to obtain export analysis configuration data; acquiring export analysis configuration data and performing service processing on the export analysis configuration data to obtain a service processing data set; and analyzing the export target file through the adaptive data processor according to the service processing data set to obtain export result data.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data importing method, a data exporting device, and a data processing system.
Background
The import and export technology supports the import into the database through the excel file and the export from the database into the excel file, and processes the data through the database language. The existing import and export technology is applied to the field of enterprise application software, but because the business of an enterprise is complex and changeable, the business data has wide sources and complex processing, and the processing process is more complex when the import and export processing of the data is carried out. In enterprise application based on metadata driving, the existing import and export technology abstracts public service attributes, so that complex and changeable service scenes can be supported; in enterprise applications based on domain driving, the existing import and export technology is modeled based on a domain model. In the multi-tenant platform service, platform services are provided for different tenants through an import and export technology, and the platform services are protected through tenant isolation and tenant resource management and control.
However, in the existing data import and export technology, only data itself is concerned in the data processing process, and the processing logic of the data does not relate to business aspects, so that the data sources supported by the technology are single, and a system is rushed when the data volume is large. Due to the fact that the scenes of enterprise application are complex and changeable, and the data sources are multiple, the existing technology is difficult to support.
Disclosure of Invention
The invention provides a data import method, a data export device and a data processing system, which are used for carrying out metadata configuration on an import/export target file and analyzing and processing the file by combining an adaptive data processor so as to solve the technical problem that the existing data import/export technology has single data source and cannot be applied to enterprise application with complicated and variable scenes, so that when the data import/export is carried out, the processing logic of the data is independent and is adaptive to the corresponding data processor, and the complicated and variable data is further processed so as to prevent the system from running.
In order to solve the above technical problem, an embodiment of the present invention provides a data importing method, including:
carrying out metadata configuration information on the imported target file;
acquiring and analyzing metadata configuration information of an import target file to obtain import analysis configuration data;
analyzing the imported target file through an adaptive data processor according to the imported analysis configuration data to obtain analysis processing file data;
and acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data.
As a preferred scheme, the performing metadata configuration information on the import destination file includes:
configuring a mapping relation between fields for the imported target file;
configuring data processing information for the imported target file;
and configuring information of the file to be imported for the import target file.
As a preferred scheme, the obtaining and analyzing metadata configuration information of an import target file includes:
acquiring metadata configuration information of the import target file;
analyzing and processing the mapping relation between the fields in the metadata configuration information;
analyzing and processing data processing information in the metadata configuration information;
and analyzing and processing the file information to be imported in the metadata configuration information.
As a preferred scheme, the parsing the import target file through the adapted data processor according to the import parsing configuration data to obtain parsing processing file data includes:
applying an import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the size of the residual permission number; if not, terminating and returning;
judging that when the number of the application import licenses is smaller than the remaining number of the licenses, acquiring the corresponding number of the licenses; otherwise, terminating and returning;
after the corresponding license number is obtained, adapting to a corresponding data processor, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
As a preferred scheme, the acquiring the file data processed by analysis and performing service processing on the file data processed by analysis to obtain import result data includes:
inputting and acquiring the analysis processing file data and acquiring service logic metadata;
carrying out conversion verification processing on the analyzed file data to obtain first import data;
performing service logic processing according to the service logic metadata, and executing the service logic to obtain second import data;
and processing the first import data and the second import data to generate import result data of an import target file, and storing the import result data in a database.
An embodiment of the present invention further provides a data importing apparatus, including:
the import data configuration module is used for carrying out metadata configuration information on the import target file;
the import analysis engine module is used for acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data;
the import data processing module is used for analyzing the import target file through the adaptive data processor according to the import analysis configuration data to obtain analysis processing file data;
and the import service engine module is used for acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data.
Preferably, the import data configuration module includes:
the first import configuration unit is used for configuring the mapping relation between the fields for the import target file;
the second import configuration unit is used for configuring data processing information for the import target file;
and the third import configuration unit is used for configuring the information of the file to be imported to the import target file.
Preferably, the import parsing engine module includes:
the import analysis acquisition unit is used for acquiring the metadata configuration information of the import target file;
the first import analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second import analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third import analysis unit is used for analyzing and processing the file information to be imported in the metadata configuration information.
Preferably, the import data processing module includes:
the first import judgment unit is used for applying import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the residual permission number; if not, terminating and returning;
the second import judgment unit is used for judging that when the application import permission number is smaller than the residual permission number, the corresponding permission number is obtained; otherwise, terminating and returning;
and the imported data processing unit is used for adapting to a corresponding data processor after the corresponding license number is obtained, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
As a preferred scheme, the importing service engine module includes:
the import data acquisition unit is used for inputting and acquiring the analysis processing file data and acquiring service logic metadata;
the import service processing unit is used for carrying out conversion and verification processing on the analyzed and processed file data to obtain first import data;
the import service logic unit is used for performing service logic processing according to the service logic metadata and executing the service logic to obtain second import data;
and the import result generating unit is used for processing the first import data and the second import data to generate import result data of an import target file and storing the import result data in a database.
The embodiment of the invention also provides a data export method, which comprises the following steps:
carrying out metadata configuration information on the export target file;
acquiring metadata configuration information of an export target file and analyzing the metadata configuration information to obtain export analysis configuration data;
acquiring the export analysis configuration data and carrying out service processing on the export analysis configuration data to obtain a service processing data set;
and analyzing the export target file through the adaptive data processor according to the service processing data set to obtain export result data.
Preferably, the performing metadata configuration information on the export target file includes:
configuring a mapping relation between fields for the export target file;
configuring data processing information for the export target file;
and configuring information of the file to be exported for the export target file.
As a preferred scheme, the obtaining and analyzing metadata configuration information of an export target file includes:
acquiring metadata configuration information of the export target file;
analyzing and processing the mapping relation between the fields in the metadata configuration information;
analyzing and processing data processing information in the metadata configuration information;
and analyzing and processing the file information to be exported in the metadata configuration information.
As a preferred scheme, the obtaining the export analysis configuration data and performing service processing on the export analysis configuration data to obtain a service processing data set includes:
inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
carrying out conversion check processing on the export analysis configuration data to obtain first export data;
performing service logic processing according to the service logic metadata, and executing the service logic to obtain second export data;
and processing the first derived data and the second derived data to generate a service processing data set.
As a preferred scheme, the analyzing the export target file through an adaptive data processor according to a service processing data set to obtain export result data includes:
applying for a export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and if so, judging the size of the application export permission number and the residual permission number; if not, terminating and returning;
judging that when the number of the application export licenses is smaller than the residual licenses, obtaining the corresponding number of the licenses; otherwise, terminating and returning;
and after the corresponding license number is obtained, adapting to a corresponding data processor, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
An embodiment of the present invention further provides a data export apparatus, including:
the export data configuration module is used for carrying out metadata configuration information on the export target file;
the export analysis engine module is used for acquiring and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data;
the export service engine module is used for acquiring the export analysis configuration data and carrying out service processing on the export analysis configuration data to obtain a service processing data set;
and the export data processing module is used for analyzing and processing the export target file through the adaptive data processor according to the service processing data set to obtain export result data.
Preferably, the export data configuration module includes:
the first export configuration unit is used for configuring the mapping relation between the fields for the export target file;
the second exports the disposition unit, is used for disposing the data handling information to exporting the goal file;
and the third export configuration unit is used for configuring the information of the file to be exported for the export target file.
Preferably, the export analysis engine module includes:
an export analysis acquisition unit for acquiring metadata configuration information of the export target file;
the first derivation analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second derivation analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third export analysis unit is used for analyzing and processing the file information to be exported in the metadata configuration information.
Preferably, the export service engine module includes:
the export data acquisition unit is used for inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
the export service processing unit is used for carrying out conversion check processing on the export analysis configuration data to obtain first export data;
the export service logic unit is used for carrying out service logic processing according to the service logic metadata and executing the service logic to obtain second export data;
and the derived data processing unit is used for processing the first derived data and the second derived data to generate a service processing data set.
Preferably, the derived data processing module includes:
the first export judgment unit is used for applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and judging the size of the application export permission number and the residual permission number if the data value of the export target file is larger than zero; if not, terminating and returning;
the second derivation judging unit is used for judging that when the application derivation license number is smaller than the residual license number, the corresponding license number is obtained; otherwise, terminating and returning;
and the export result generating unit is used for adapting to the corresponding data processor after the corresponding license number is obtained, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
An embodiment of the present invention further provides a data processing system, including:
the metadata configuration module comprises an import data configuration module and an export data configuration module; the import data configuration module is used for carrying out metadata configuration information on an import target file; the export data configuration module is used for carrying out metadata configuration information on the export target file;
the import analysis engine module is used for acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data;
the export analysis engine module is used for acquiring and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data;
the data processing module comprises an import data processing module and an export data processing module; the imported data processing module is used for analyzing the imported target file through an adaptive data processor according to the imported analysis configuration data to obtain analysis processing file data; the export data processing module is used for analyzing and processing the export target file through an adaptive data processor according to a service processing data set to obtain export result data;
the business engine module comprises an import business engine module and an export business engine module; the import service engine module is used for acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data; and the export service engine module is used for acquiring the export analysis configuration data and performing service processing on the export analysis configuration data to obtain a service processing data set.
Preferably, the import data configuration module includes:
the first import configuration unit is used for configuring the mapping relation between the fields for the import target file;
the second import configuration unit is used for configuring data processing information for the import target file;
and the third import configuration unit is used for configuring the information of the file to be imported to the import target file.
Preferably, the import parsing engine module includes:
the import analysis acquisition unit is used for acquiring the metadata configuration information of the import target file;
the first import analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second import analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third import analysis unit is used for analyzing and processing the file information to be imported in the metadata configuration information.
Preferably, the import data processing module includes:
the first import judgment unit is used for applying import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the residual permission number; if not, terminating and returning;
the second import judgment unit is used for judging that when the application import permission number is smaller than the residual permission number, the corresponding permission number is obtained; otherwise, terminating and returning;
and the imported data processing unit is used for adapting to a corresponding data processor after the corresponding license number is obtained, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
As a preferred scheme, the importing service engine module includes:
the import data acquisition unit is used for inputting and acquiring the analysis processing file data and acquiring service logic metadata;
the import service processing unit is used for carrying out conversion and verification processing on the analyzed and processed file data to obtain first import data;
the import service logic unit is used for performing service logic processing according to the service logic metadata and executing the service logic to obtain second import data;
and the import result generating unit is used for processing the first import data and the second import data to generate import result data of an import target file and storing the import result data in a database.
Preferably, the export data configuration module includes:
the first export configuration unit is used for configuring the mapping relation between the fields for the export target file;
the second exports the disposition unit, is used for disposing the data handling information to exporting the goal file;
and the third export configuration unit is used for configuring the information of the file to be exported for the export target file.
Preferably, the export analysis engine module includes:
an export analysis acquisition unit for acquiring metadata configuration information of the export target file;
the first derivation analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second derivation analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third export analysis unit is used for analyzing and processing the file information to be exported in the metadata configuration information.
Preferably, the export service engine module includes:
the export data acquisition unit is used for inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
the export service processing unit is used for carrying out conversion check processing on the export analysis configuration data to obtain first export data;
the export service logic unit is used for carrying out service logic processing according to the service logic metadata and executing the service logic to obtain second export data;
and the derived data processing unit is used for processing the first derived data and the second derived data to generate a service processing data set.
Preferably, the derived data processing module includes:
the first export judgment unit is used for applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and judging the size of the application export permission number and the residual permission number if the data value of the export target file is larger than zero; if not, terminating and returning;
the second derivation judging unit is used for judging that when the application derivation license number is smaller than the residual license number, the corresponding license number is obtained; otherwise, terminating and returning;
and the export result generating unit is used for adapting to the corresponding data processor after the corresponding license number is obtained, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
according to the invention, the metadata configuration is carried out on the import/export target file, and the file is analyzed and processed by combining the adaptive data processor, so that the technical problem that the existing data import/export technology has a single data source and cannot be applied to enterprise applications with complex and changeable scenes is solved, and therefore, when data import/export is carried out, the processing logic of the data is independent and is adaptive to the corresponding data processor, and the complex and changeable data is further processed, so that the system is prevented from running.
Drawings
FIG. 1: a flow chart of a data importing method in the embodiment of the invention;
FIG. 2: a flow chart of a data export method in an embodiment of the invention;
FIG. 3: a flow chart for data processing for application permission in the embodiment of the present invention;
FIG. 4: a flow chart of a business engine module processing process in the embodiment of the invention;
FIG. 5: the analysis processing flow chart of the import and export analysis engine module in the embodiment of the invention is shown;
FIG. 6: is a data processing system architecture diagram in an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a preferred embodiment of the present invention provides a data importing method, including:
s1, carrying out metadata configuration information on the import target file; in this embodiment, the step S1 includes: s11, configuring the mapping relation between the fields for the import target file; s12, configuring data processing information for the import target file; s13, the file information to be imported is configured for the import target file.
The whole process of importing excel file data into the database is shown in the example. Suppose there is a file "import contest. xlsx" with the following data, see table 1 below.
| Contest code | Name of contest | Competitive products brand |
| COM00000001 | Lehu amino acid vitamin functional beverage | Happy tiger |
| COM00000002 | Natural mineral water is introduced into Baisuishan | Baisui mountain |
Table 1: importing a target file
The metadata configuration information for performing import behavior mainly includes: mapping relation between fields, data processing information, information of files to be imported and the like.
Referring to table 2, a mapping relationship between the column name of the excel table and the service entity attribute is configured:
| name of business entity | Properties | Name of file column |
| Competition product | code | Contest code |
| Competition product | name | Name of contest |
| Dictionary | dicvalue | Competitive products brand |
Table 2: configuration mapping relation processing result table
Referring to table 3, data processing information is configured:
| properties | Value of |
| Business logic | Newly-increased competitive products |
| Whether to process in sequence | Is that |
| Beginning line of data | 1 |
| Duplicate row processing strategy | Skip over |
Table 3: configuration data processing information result table
Referring to table 4, the information of the file to be imported is configured:
| properties | Value of |
| Filename | Lead-in contest. xlsx |
| File type | excel |
| File path | att/cef/cef3-7e88-4c9e-894 d/lead-in competitive product xlsx |
Table 4: configuring result table of information of file to be imported
S2, acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data; in this embodiment, the step S2 includes: s21, acquiring the metadata configuration information of the import target file; s22, analyzing and processing the mapping relation between the fields in the metadata configuration information; s23, analyzing and processing the data processing information in the metadata configuration information; and S24, analyzing and processing the file information to be imported in the metadata configuration information.
Referring to fig. 5, it is necessary to first obtain metadata configuration information of an import behavior, and process mapping relationships, data processing information, and file information to be imported step by step, so as to finally obtain data in the following format, as shown in tables 5 and 6 below.
| Properties | Value of |
| Filename | Lead-in contest. xlsx |
| File type | excel |
| File path | att/cef/cef3-7e88-4c9e-894 d/lead-in competitive product xlsx |
| Business logic | Newly-increased competitive products |
| Whether to process in sequence | Is that |
| Beginning line of data | 1 |
| Duplicate row processing strategy | Skip over |
| List of names | "Bingping code", "Bingping code" and "Bingping brand"] |
| Field list | [“code”,“name”,“dicvalue”] |
Table 5: import analysis configuration data table one
Table 6: import analysis configuration data table two
S3, analyzing the import target file through the adaptive data processor according to the import analysis configuration data to obtain analysis processing file data; in this embodiment, the step S3 includes: s31, according to the size of the tenant and the data, applying for the import permission number, judging whether the data value of the import target file is larger than zero, if so, judging the size of the application import permission number and the remaining permission number; if not, terminating and returning; s32, judging that when the number of the application import licenses is less than the residual license number, obtaining the corresponding license number; otherwise, terminating and returning; and S33, after the corresponding license number is obtained, adapting to a corresponding data processor, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
As shown in fig. 3, the import permission number needs to be applied before the data is analyzed: the permission number is imported according to the size application of the tenant and the data, and if the permission number is insufficient, the operation is terminated and returned; if a sufficient number of licenses is applied, the corresponding data processor is adapted, and file analysis and preliminary verification of the data are performed according to the data analyzed in the step S2.
For example, the excel file processor is used to parse the data in the table to obtain the data in the following format, as shown in table 7 below.
| Line number | Properties | Value of |
| 1 | code | COM00000001 |
| 1 | name | Lehu amino acid vitamin functional beverage |
| 1 | dicvalue | Happy tiger |
| 2 | code | COM00000002 |
| 2 | name | Natural mineral water is introduced into Baisuishan |
| 2 | dicvalue | Baisui mountain |
Table 7: parsing a processed file data table
And S4, acquiring the analysis processing file data and carrying out service processing on the analysis processing file data to obtain import result data. In this embodiment, the step S4 includes: s41, inputting and acquiring the analysis processing file data and acquiring service logic metadata; s42, carrying out conversion verification processing on the file data subjected to analysis processing to obtain first import data; s43, performing service logic processing according to the service logic metadata, and executing the service logic to obtain second import data; and S44, processing the first import data and the second import data to generate import result data of an import target file, and storing the import result data in a database.
And (4) performing service processing on the data, as shown in fig. 4, that is, uploading the data obtained in step S3 to the step line by line for processing, acquiring metadata configuration information of "newly added auction product" logic, then converting and verifying the input data, performing the processing after processing the service logic, and performing final processing after obtaining the data.
The input data is converted, for example, the first line data is in the following format, as shown in table 8.
| Name of business entity | Properties | Value of |
| Competition product | code | COM00000001 |
| Competition product | name | Lehu amino acid vitamin functional beverage |
| Dictionary | dicvalue | Happy tiger |
Table 8: data conversion to generate data table
The input data is finally processed and stored in the following table of the database, please refer to table 9 and table 10.
| id | code | name | brand |
| 1015062751866916864 | COM00000001 | Lehu amino acid vitamin functional beverage | 989423172740845568 |
| 1015062754127646720 | COM00000002 | Natural mineral water is introduced into Baisuishan | 989423141124182016 |
Table 9: first import data generating data table
| dictionaryid | dickey | dicvalue |
| 989423172740845568 | 989423172740845568 | Happy tiger |
| 989423141124182016 | 989423141124182016 | Baisui mountain |
Table 10: second import data generating data table
Correspondingly, an embodiment of the present invention further provides a data importing apparatus, including:
the import data configuration module is used for carrying out metadata configuration information on the import target file;
the import analysis engine module is used for acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data;
the import data processing module is used for analyzing the import target file through the adaptive data processor according to the import analysis configuration data to obtain analysis processing file data;
and the import service engine module is used for acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data.
In this embodiment, the importing data configuration module includes:
the first import configuration unit is used for configuring the mapping relation between the fields for the import target file;
the second import configuration unit is used for configuring data processing information for the import target file;
and the third import configuration unit is used for configuring the information of the file to be imported to the import target file.
In this embodiment, the importing parsing engine module includes:
the import analysis acquisition unit is used for acquiring the metadata configuration information of the import target file;
the first import analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second import analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third import analysis unit is used for analyzing and processing the file information to be imported in the metadata configuration information.
In this embodiment, the importing data processing module includes:
the first import judgment unit is used for applying import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the residual permission number; if not, terminating and returning;
the second import judgment unit is used for judging that when the application import permission number is smaller than the residual permission number, the corresponding permission number is obtained; otherwise, terminating and returning;
and the imported data processing unit is used for adapting to a corresponding data processor after the corresponding license number is obtained, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
In this embodiment, the importing service engine module includes:
the import data acquisition unit is used for inputting and acquiring the analysis processing file data and acquiring service logic metadata;
the import service processing unit is used for carrying out conversion and verification processing on the analyzed and processed file data to obtain first import data;
the import service logic unit is used for performing service logic processing according to the service logic metadata and executing the service logic to obtain second import data;
and the import result generating unit is used for processing the first import data and the second import data to generate import result data of an import target file and storing the import result data in a database.
Referring to fig. 2, a preferred embodiment of the present invention provides a data export method, including:
s1, carrying out metadata configuration information on the export target file; in this embodiment, the step S1 includes: s11, mapping relation between fields is configured for the export target file; s12, configuring data processing information for the export target file; s13, the file information to be exported is configured for the export target file.
This example shows the whole process of exporting CSV files from the competitive product data, and it is assumed that the CSV files of competitive products need to be exported, and the following data exists in the database, please refer to table 11 and table 12.
| id | code | name | brand |
| 1015062751866916864 | COM00000001 | Lehu amino acid vitamin functional beverage | 989423172740845568 |
| 1015062754127646720 | COM00000002 | Natural mineral water is introduced into Baisuishan | 989423141124182016 |
Table 11: export target File one in data export embodiments
| dictionaryid | dickey | dicvalue |
| 989423172740845568 | 989423172740845568 | Happy tiger |
| 989423141124182016 | 989423141124182016 | Baisui mountain |
Table 12: export target File two in data export embodiments
The metadata configuration information for exporting behavior mainly includes mapping relationship between fields, data processing information, export file information, and the like.
For example, configure the mapping relationship between the service entity attribute and the CSV file header, please refer to table 13:
| name of business entity | Properties | Name of file column |
| Competition product | code | Contest code |
| Competition product | name | Name of contest |
| Competition product | brand | Competitive products brand |
Table 13: exporting example configuration mapping relation table
Configure data processing information, please refer to table 14:
| properties | Value of |
| Business logic | Bidding list query |
Table 14: exporting example configuration data processing information tables
Export file information is configured, please refer to table 15:
| properties | Value of |
| Filename | Derived contestants csv |
| File type | CSV |
Table 15: export example configuration export File information Table
S2, obtaining and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data; in this embodiment, the step S2 includes: s21, acquiring metadata configuration information of the export target file; s22, analyzing and processing the mapping relation between the fields in the metadata configuration information; s23, analyzing and processing the data processing information in the metadata configuration information; and S24, analyzing and processing the file information to be exported in the metadata configuration information.
As shown in fig. 5, it is necessary to first obtain metadata configuration information of export behavior, and process the mapping relationship, data processing information, and export file information step by step, and finally obtain data in the following format, please refer to table 16.
| Properties | Value of |
| Filename | Derived contestants csv |
| File type | CSV |
| File path | att/ase/ase4-629s-4a2s-354 e/export bid csv |
| List of names | "Bingping code", "Bingping name", "Bingping brand"] |
| Field list | [“code”,“name”,“brand”] |
| Business logic | Bidding list query |
Table 16: exporting parsing configuration data table
S3, acquiring the export analysis configuration data and carrying out service processing on the export analysis configuration data to obtain a service processing data set; in this embodiment, the step S3 includes: s31, inputting and acquiring the export analysis configuration data and acquiring service logic metadata; s32, carrying out conversion check processing on the export analysis configuration data to obtain first export data; s33, performing service logic processing according to the service logic metadata, and executing the service logic to obtain second export data; and S34, processing the first derived data and the second derived data to generate a service processing data set.
As shown in fig. 4, the following operations are performed after the data obtained in step S2 is uploaded to this step: firstly, acquiring metadata information of 'competitive product list query' service logic, and processing input parameters; and then processing the service logic and executing the service logic, and finally processing the service logic after obtaining a result.
For example, after the bid table is queried, the value of the field brand is processed and replaced with the value corresponding to the dicvalue field in the dictionary table, so as to finally obtain the following data set, please refer to table 17.
Table 17: business processing data set table
And S4, analyzing the export target file through the adaptive data processor according to the service processing data set to obtain export result data. In this embodiment, the step S4 includes: s41, applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, if so, judging the size of the application export permission number and the residual permission number; if not, terminating and returning; s42, judging that when the number of the export licenses is smaller than the residual license number, obtaining the corresponding license number; otherwise, terminating and returning; s43, after the corresponding license number is obtained, adapting to the corresponding data processor, and analyzing and data checking the export target file according to the service processing data set to obtain export result data.
As shown in fig. 3, before data processing, a derived license number needs to be applied: the permission number is derived according to the size application of the tenant and the data, and if the permission number is insufficient, the return is terminated; if a sufficient number of licenses is applied, the data processor is adapted to process the data set obtained in step S3.
Such as: the CSV file content is generated as follows, the title of the first row is generated from the list of column names of step S2, and the data with the index 1 in step S3 is converted into the following from the list of fields of step S2: [ "COM00000001", "Lehu amino acid vitamin functional beverage", "Lehu" ], and writes the same into a file, and similarly processes data with a subscript of 2 to obtain: "Bingping code", "Bingping name", "Bingping brand", "COM00000001", "Hakko amino acid vitamin functional beverage", "Hakko", "COM00000002", "Baisui mountain quoted natural mineral water" and "Baisui mountain".
Correspondingly, an embodiment of the present invention further provides a data export apparatus, including:
the export data configuration module is used for carrying out metadata configuration information on the export target file;
the export analysis engine module is used for acquiring and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data;
the export service engine module is used for acquiring the export analysis configuration data and carrying out service processing on the export analysis configuration data to obtain a service processing data set;
and the export data processing module is used for analyzing and processing the export target file through the adaptive data processor according to the service processing data set to obtain export result data.
In this embodiment, the export data configuration module includes:
the first export configuration unit is used for configuring the mapping relation between the fields for the export target file;
the second exports the disposition unit, is used for disposing the data handling information to exporting the goal file;
and the third export configuration unit is used for configuring the information of the file to be exported for the export target file.
In this embodiment, the export parsing engine module includes:
an export analysis acquisition unit for acquiring metadata configuration information of the export target file;
the first derivation analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second derivation analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third export analysis unit is used for analyzing and processing the file information to be exported in the metadata configuration information.
In this embodiment, the export service engine module includes:
the export data acquisition unit is used for inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
the export service processing unit is used for carrying out conversion check processing on the export analysis configuration data to obtain first export data;
the export service logic unit is used for carrying out service logic processing according to the service logic metadata and executing the service logic to obtain second export data;
and the derived data processing unit is used for processing the first derived data and the second derived data to generate a service processing data set.
In this embodiment, the derived data processing module includes:
the first export judgment unit is used for applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and judging the size of the application export permission number and the residual permission number if the data value of the export target file is larger than zero; if not, terminating and returning;
the second derivation judging unit is used for judging that when the application derivation license number is smaller than the residual license number, the corresponding license number is obtained; otherwise, terminating and returning;
and the export result generating unit is used for adapting to the corresponding data processor after the corresponding license number is obtained, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
Referring to fig. 6, an embodiment of the present invention further provides a data processing system, where the system can implement data import and data export, and the system includes:
the metadata configuration module comprises an import data configuration module and an export data configuration module; the import data configuration module is used for carrying out metadata configuration information on an import target file; the export data configuration module is used for carrying out metadata configuration information on the export target file;
the import analysis engine module is used for acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data;
the export analysis engine module is used for acquiring and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data;
the data processing module comprises an import data processing module and an export data processing module; the imported data processing module is used for analyzing the imported target file through an adaptive data processor according to the imported analysis configuration data to obtain analysis processing file data; the export data processing module is used for analyzing and processing the export target file through an adaptive data processor according to a service processing data set to obtain export result data;
the business engine module comprises an import business engine module and an export business engine module; the import service engine module is used for acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data; and the export service engine module is used for acquiring the export analysis configuration data and performing service processing on the export analysis configuration data to obtain a service processing data set.
In this embodiment, the importing data configuration module includes:
the first import configuration unit is used for configuring the mapping relation between the fields for the import target file;
the second import configuration unit is used for configuring data processing information for the import target file;
and the third import configuration unit is used for configuring the information of the file to be imported to the import target file.
In this embodiment, the importing parsing engine module includes:
the import analysis acquisition unit is used for acquiring the metadata configuration information of the import target file;
the first import analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second import analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third import analysis unit is used for analyzing and processing the file information to be imported in the metadata configuration information.
In this embodiment, the importing data processing module includes:
the first import judgment unit is used for applying import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the residual permission number; if not, terminating and returning;
the second import judgment unit is used for judging that when the application import permission number is smaller than the residual permission number, the corresponding permission number is obtained; otherwise, terminating and returning;
and the imported data processing unit is used for adapting to a corresponding data processor after the corresponding license number is obtained, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
In this embodiment, the importing service engine module includes:
the import data acquisition unit is used for inputting and acquiring the analysis processing file data and acquiring service logic metadata;
the import service processing unit is used for carrying out conversion and verification processing on the analyzed and processed file data to obtain first import data;
the import service logic unit is used for performing service logic processing according to the service logic metadata and executing the service logic to obtain second import data;
and the import result generating unit is used for processing the first import data and the second import data to generate import result data of an import target file and storing the import result data in a database.
In this embodiment, the export data configuration module includes:
the first export configuration unit is used for configuring the mapping relation between the fields for the export target file;
the second exports the disposition unit, is used for disposing the data handling information to exporting the goal file;
and the third export configuration unit is used for configuring the information of the file to be exported for the export target file.
In this embodiment, the export parsing engine module includes:
an export analysis acquisition unit for acquiring metadata configuration information of the export target file;
the first derivation analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second derivation analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third export analysis unit is used for analyzing and processing the file information to be exported in the metadata configuration information.
In this embodiment, the export service engine module includes:
the export data acquisition unit is used for inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
the export service processing unit is used for carrying out conversion check processing on the export analysis configuration data to obtain first export data;
the export service logic unit is used for carrying out service logic processing according to the service logic metadata and executing the service logic to obtain second export data;
and the derived data processing unit is used for processing the first derived data and the second derived data to generate a service processing data set.
In this embodiment, the derived data processing module includes:
the first export judgment unit is used for applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and judging the size of the application export permission number and the residual permission number if the data value of the export target file is larger than zero; if not, terminating and returning;
the second derivation judging unit is used for judging that when the application derivation license number is smaller than the residual license number, the corresponding license number is obtained; otherwise, terminating and returning;
and the export result generating unit is used for adapting to the corresponding data processor after the corresponding license number is obtained, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
Based on the above embodiments, the data processing system scheme is composed of a metadata configuration module, an import analysis engine module, an export analysis engine module, a business engine module, and a data processing module.
(1) The metadata configuration module: the metadata configuration information for managing data import and export mainly comprises a mapping relation between file data and business object attributes, a processing rule of data exception, business logic information and the like.
(2) And importing an analysis engine module: and acquiring the imported metadata configuration information from the metadata configuration library for analysis.
(3) And a export analysis engine module: and acquiring the derived metadata configuration information from the metadata configuration library for analysis.
(4) And a business engine module: after the configuration of the analysis metadata of the analysis engine module is imported, the data generated by the data processing module is transmitted into the service engine module for service processing; and the export analysis engine module analyzes the metadata configuration information and then enters the service engine module for service processing to obtain a data set for the data processing module to process.
(5) And the data processing module: the method is suitable for various data processors, supports the preliminary verification of data, supports data from different sources, supports different types of file operation and the like.
When data is imported and exported, the processing logic of the data is modeled based on the domain model and is independent, so that the scenes of wide application sources and complex business of enterprises are met; the system can be adapted to various data processors and can support data from different sources, files of different types and the like; the data import and export of multiple tenants are supported, and the normal operation of the system is ensured by controlling the resources of the tenants.
The above-mentioned embodiments are provided to further explain the objects, technical solutions and advantages of the present invention in detail, and it should be understood that the above-mentioned embodiments are only examples of the present invention and are not intended to limit the scope of the present invention. It should be understood that any modifications, equivalents, improvements and the like, which come within the spirit and principle of the invention, may occur to those skilled in the art and are intended to be included within the scope of the invention.
Claims (23)
1. A data import method, comprising:
carrying out metadata configuration information on the imported target file;
acquiring and analyzing metadata configuration information of an import target file to obtain import analysis configuration data;
analyzing the imported target file through an adaptive data processor according to the imported analysis configuration data to obtain analysis processing file data;
acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data;
the acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data includes:
inputting and acquiring the analysis processing file data and acquiring service logic metadata;
carrying out conversion verification processing on the analyzed file data to obtain first import data;
performing service logic processing according to the service logic metadata, and executing the service logic to obtain second import data;
and processing the first import data and the second import data to generate import result data of an import target file, and storing the import result data in a database.
2. The data import method according to claim 1, wherein the performing metadata configuration information on the import target file includes:
configuring a mapping relation between fields for the imported target file;
configuring data processing information for the imported target file;
and configuring information of the file to be imported for the import target file.
3. The data import method according to claim 2, wherein the obtaining and parsing the metadata configuration information of the import destination file includes:
acquiring metadata configuration information of the import target file;
analyzing and processing the mapping relation between the fields in the metadata configuration information;
analyzing and processing data processing information in the metadata configuration information;
and analyzing and processing the file information to be imported in the metadata configuration information.
4. The data import method according to claim 1, wherein the parsing the import target file according to the import parsing configuration data by using an adapted data processor to obtain parsing file data includes:
applying an import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the size of the residual permission number; if not, terminating and returning;
judging that when the number of the application import licenses is smaller than the remaining number of the licenses, acquiring the corresponding number of the licenses; otherwise, terminating and returning;
after the corresponding license number is obtained, adapting to a corresponding data processor, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
5. A data importing apparatus, comprising:
the import data configuration module is used for carrying out metadata configuration information on the import target file;
the import analysis engine module is used for acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data;
the import data processing module is used for analyzing the import target file through the adaptive data processor according to the import analysis configuration data to obtain analysis processing file data;
the import service engine module is used for acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data;
the import service engine module comprises:
the import data acquisition unit is used for inputting and acquiring the analysis processing file data and acquiring service logic metadata;
the import service processing unit is used for carrying out conversion and verification processing on the analyzed and processed file data to obtain first import data;
the import service logic unit is used for performing service logic processing according to the service logic metadata and executing the service logic to obtain second import data;
and the import result generating unit is used for processing the first import data and the second import data to generate import result data of an import target file and storing the import result data in a database.
6. The data import apparatus of claim 5, wherein the import data configuration module comprises:
the first import configuration unit is used for configuring the mapping relation between the fields for the import target file;
the second import configuration unit is used for configuring data processing information for the import target file;
and the third import configuration unit is used for configuring the information of the file to be imported to the import target file.
7. The data import apparatus of claim 6, wherein the import parsing engine module comprises:
the import analysis acquisition unit is used for acquiring the metadata configuration information of the import target file;
the first import analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second import analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third import analysis unit is used for analyzing and processing the file information to be imported in the metadata configuration information.
8. The data import apparatus of claim 5, wherein the import data processing module comprises:
the first import judgment unit is used for applying import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the residual permission number; if not, terminating and returning;
the second import judgment unit is used for judging that when the application import permission number is smaller than the residual permission number, the corresponding permission number is obtained; otherwise, terminating and returning;
and the imported data processing unit is used for adapting to a corresponding data processor after the corresponding license number is obtained, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
9. A method of data derivation, comprising:
carrying out metadata configuration information on the export target file;
acquiring metadata configuration information of an export target file and analyzing the metadata configuration information to obtain export analysis configuration data;
acquiring the export analysis configuration data and carrying out service processing on the export analysis configuration data to obtain a service processing data set;
analyzing the export target file through an adaptive data processor according to the service processing data set to obtain export result data;
the obtaining the export analysis configuration data and performing service processing on the export analysis configuration data to obtain a service processing data set includes:
inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
carrying out conversion check processing on the export analysis configuration data to obtain first export data;
performing service logic processing according to the service logic metadata, and executing the service logic to obtain second export data;
and processing the first derived data and the second derived data to generate a service processing data set.
10. The data export method of claim 9, wherein said performing metadata configuration information on the export target file comprises:
configuring a mapping relation between fields for the export target file;
configuring data processing information for the export target file;
and configuring information of the file to be exported for the export target file.
11. The data export method of claim 10, wherein the obtaining and parsing the metadata configuration information of the export target file comprises:
acquiring metadata configuration information of the export target file;
analyzing and processing the mapping relation between the fields in the metadata configuration information;
analyzing and processing data processing information in the metadata configuration information;
and analyzing and processing the file information to be exported in the metadata configuration information.
12. The data export method of claim 9, wherein the parsing the export target file according to the business process data set by the adapted data processor to obtain export result data comprises:
applying for a export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and if so, judging the size of the application export permission number and the residual permission number; if not, terminating and returning;
judging that when the number of the application export licenses is smaller than the residual licenses, obtaining the corresponding number of the licenses; otherwise, terminating and returning;
and after the corresponding license number is obtained, adapting to a corresponding data processor, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
13. A data derivation apparatus, comprising:
the export data configuration module is used for carrying out metadata configuration information on the export target file;
the export analysis engine module is used for acquiring and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data;
the export service engine module is used for acquiring the export analysis configuration data and carrying out service processing on the export analysis configuration data to obtain a service processing data set;
the export data processing module is used for analyzing and processing the export target file through the adaptive data processor according to the service processing data set to obtain export result data;
the export service engine module comprises:
the export data acquisition unit is used for inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
the export service processing unit is used for carrying out conversion check processing on the export analysis configuration data to obtain first export data;
the export service logic unit is used for carrying out service logic processing according to the service logic metadata and executing the service logic to obtain second export data;
and the derived data processing unit is used for processing the first derived data and the second derived data to generate a service processing data set.
14. The data export apparatus of claim 13, wherein the export data configuration module comprises:
the first export configuration unit is used for configuring the mapping relation between the fields for the export target file;
the second exports the disposition unit, is used for disposing the data handling information to exporting the goal file;
and the third export configuration unit is used for configuring the information of the file to be exported for the export target file.
15. The data exporting apparatus according to claim 14, wherein the export parsing engine module comprises:
an export analysis acquisition unit for acquiring metadata configuration information of the export target file;
the first derivation analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second derivation analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third export analysis unit is used for analyzing and processing the file information to be exported in the metadata configuration information.
16. The data export apparatus of claim 13, wherein the export data processing module comprises:
the first export judgment unit is used for applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and judging the size of the application export permission number and the residual permission number if the data value of the export target file is larger than zero; if not, terminating and returning;
the second derivation judging unit is used for judging that when the application derivation license number is smaller than the residual license number, the corresponding license number is obtained; otherwise, terminating and returning;
and the export result generating unit is used for adapting to the corresponding data processor after the corresponding license number is obtained, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
17. A data processing system, comprising:
the metadata configuration module comprises an import data configuration module and an export data configuration module; the import data configuration module is used for carrying out metadata configuration information on an import target file; the export data configuration module is used for carrying out metadata configuration information on the export target file;
the import analysis engine module is used for acquiring and analyzing the metadata configuration information of the import target file to obtain import analysis configuration data;
the export analysis engine module is used for acquiring and analyzing the metadata configuration information of the export target file to obtain export analysis configuration data;
the data processing module comprises an import data processing module and an export data processing module; the imported data processing module is used for analyzing the imported target file through an adaptive data processor according to the imported analysis configuration data to obtain analysis processing file data; the export data processing module is used for analyzing and processing the export target file through an adaptive data processor according to a service processing data set to obtain export result data;
the business engine module comprises an import business engine module and an export business engine module; the import service engine module is used for acquiring the analysis processing file data and performing service processing on the analysis processing file data to obtain import result data; the export service engine module is used for acquiring the export analysis configuration data and performing service processing on the export analysis configuration data to obtain a service processing data set;
the import service engine module comprises:
the import data acquisition unit is used for inputting and acquiring the analysis processing file data and acquiring service logic metadata;
the import service processing unit is used for carrying out conversion and verification processing on the analyzed and processed file data to obtain first import data;
the import service logic unit is used for performing service logic processing according to the service logic metadata and executing the service logic to obtain second import data;
an import result generation unit, configured to process the first import data and the second import data to generate import result data of an import target file, and store the import result data in a database;
the export service engine module comprises:
the export data acquisition unit is used for inputting and acquiring the export analysis configuration data and acquiring service logic metadata;
the export service processing unit is used for carrying out conversion check processing on the export analysis configuration data to obtain first export data;
the export service logic unit is used for carrying out service logic processing according to the service logic metadata and executing the service logic to obtain second export data;
and the derived data processing unit is used for processing the first derived data and the second derived data to generate a service processing data set.
18. The data processing system of claim 17, wherein the importing data configuration module comprises:
the first import configuration unit is used for configuring the mapping relation between the fields for the import target file;
the second import configuration unit is used for configuring data processing information for the import target file;
and the third import configuration unit is used for configuring the information of the file to be imported to the import target file.
19. The data processing system of claim 18, wherein the import parsing engine module comprises:
the import analysis acquisition unit is used for acquiring the metadata configuration information of the import target file;
the first import analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second import analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third import analysis unit is used for analyzing and processing the file information to be imported in the metadata configuration information.
20. The data processing system of claim 17, wherein the importing data processing module comprises:
the first import judgment unit is used for applying import permission number according to the size of the tenant and the data, judging whether the data value of the import target file is larger than zero, and if so, judging the size of the application import permission number and the residual permission number; if not, terminating and returning;
the second import judgment unit is used for judging that when the application import permission number is smaller than the residual permission number, the corresponding permission number is obtained; otherwise, terminating and returning;
and the imported data processing unit is used for adapting to a corresponding data processor after the corresponding license number is obtained, and analyzing and data checking the imported target file according to the imported analysis configuration data to obtain analysis processing file data.
21. The data processing system of claim 17, wherein the export data configuration module comprises:
the first export configuration unit is used for configuring the mapping relation between the fields for the export target file;
the second exports the disposition unit, is used for disposing the data handling information to exporting the goal file;
and the third export configuration unit is used for configuring the information of the file to be exported for the export target file.
22. The data processing system of claim 21, wherein the export resolution engine module comprises:
an export analysis acquisition unit for acquiring metadata configuration information of the export target file;
the first derivation analysis unit is used for analyzing and processing the mapping relation between the fields in the metadata configuration information;
the second derivation analysis unit is used for analyzing and processing the data processing information in the metadata configuration information;
and the third export analysis unit is used for analyzing and processing the file information to be exported in the metadata configuration information.
23. The data processing system of claim 17, wherein the export data processing module comprises:
the first export judgment unit is used for applying export permission number according to the size of the tenant and the data, judging whether the data value of the export target file is larger than zero, and judging the size of the application export permission number and the residual permission number if the data value of the export target file is larger than zero; if not, terminating and returning;
the second derivation judging unit is used for judging that when the application derivation license number is smaller than the residual license number, the corresponding license number is obtained; otherwise, terminating and returning;
and the export result generating unit is used for adapting to the corresponding data processor after the corresponding license number is obtained, and analyzing and verifying the export target file according to the service processing data set to obtain export result data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910919000.4A CN110795387B (en) | 2019-09-26 | 2019-09-26 | Data import method, data export device and data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910919000.4A CN110795387B (en) | 2019-09-26 | 2019-09-26 | Data import method, data export device and data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110795387A CN110795387A (en) | 2020-02-14 |
| CN110795387B true CN110795387B (en) | 2020-11-03 |
Family
ID=69439899
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910919000.4A Active CN110795387B (en) | 2019-09-26 | 2019-09-26 | Data import method, data export device and data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110795387B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111427863B (en) * | 2020-03-11 | 2023-08-08 | 支付宝(杭州)信息技术有限公司 | Data migration method, device and equipment based on domain model |
| CN112115105A (en) * | 2020-09-28 | 2020-12-22 | 中国建设银行股份有限公司 | Service processing method, device and equipment |
| CN112835901A (en) * | 2021-02-01 | 2021-05-25 | 长沙市到家悠享家政服务有限公司 | File storage method and device, computer equipment and computer readable storage medium |
| CN112965740B (en) * | 2021-02-08 | 2024-02-09 | 中国工商银行股份有限公司 | Asset information export method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484805A (en) * | 2016-09-22 | 2017-03-08 | 联动优势电子商务有限公司 | A kind of data export method and data handling system |
| CN109241358A (en) * | 2018-08-14 | 2019-01-18 | 中国平安财产保险股份有限公司 | Metadata management method, device, computer equipment and storage medium |
| CN110019456A (en) * | 2017-11-01 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Data lead-in method, device and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW200417895A (en) * | 2003-03-12 | 2004-09-16 | Hon Hai Prec Ind Co Ltd | System and method for monitoring import and export |
| CN102456051A (en) * | 2010-10-27 | 2012-05-16 | 北京大学 | Methods and devices for importing/exporting database data |
| CN103092993B (en) * | 2013-02-18 | 2016-07-06 | 五八同城信息技术有限公司 | Data export method and device |
| US10474666B2 (en) * | 2016-10-12 | 2019-11-12 | Bank Of America Corporation | Metadata validation tool |
| CN108345682A (en) * | 2018-03-02 | 2018-07-31 | 弘成科技发展有限公司 | Platform and method are imported and exported based on what multi-tenant can configure |
-
2019
- 2019-09-26 CN CN201910919000.4A patent/CN110795387B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484805A (en) * | 2016-09-22 | 2017-03-08 | 联动优势电子商务有限公司 | A kind of data export method and data handling system |
| CN110019456A (en) * | 2017-11-01 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Data lead-in method, device and system |
| CN109241358A (en) * | 2018-08-14 | 2019-01-18 | 中国平安财产保险股份有限公司 | Metadata management method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110795387A (en) | 2020-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110795387B (en) | Data import method, data export device and data processing system | |
| CN108536761B (en) | Report data query method and server | |
| US9928251B2 (en) | System and method for distributed categorization | |
| US10169437B2 (en) | Triplestore replicator | |
| CN108228166B (en) | Template-based back-end code generation method and system | |
| US8819064B2 (en) | Method and system for data provenance management in multi-layer systems | |
| CN105447099A (en) | Log structured information extraction method and apparatus | |
| US9619537B2 (en) | Converting data objects from single- to multi-source database environment | |
| CN105530272A (en) | Method and device for application data synchronization | |
| CN110598993B (en) | Data processing method and device | |
| CN115827084A (en) | Data processing method, device, equipment and storage medium | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| US20140095527A1 (en) | Expanding high level queries | |
| CN111523030B (en) | Newspaper disc information recommendation method and device and computer readable storage medium | |
| US8224797B2 (en) | System and method for validating data record | |
| CA3203549A1 (en) | Unified verification method, device, equipment and storage medium | |
| CN1567829A (en) | General purpose data file conversion method | |
| CN110472127A (en) | A kind of data query method and system | |
| US20230035551A1 (en) | Multiple source audit log generation | |
| CN102486731A (en) | Method, device and system for enhancing visualization of software call stack of software | |
| CN110851446B (en) | Data table generation method and device, computer equipment and storage medium | |
| CN111611056A (en) | Data processing method and device, computer equipment and storage medium | |
| CN110309062A (en) | Case generation method, device, electronic equipment and storage medium | |
| US20230029899A1 (en) | Context-dependent event cleaning and publication | |
| US11616744B2 (en) | Context-dependent message extraction and transformation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: 510000 room 23bd, No. 109, TIYU West Road, Tianhe District, Guangzhou City, Guangdong Province Patentee after: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. Address before: 32B, no.103b, TianLiHe Road, Guangzhou, 510000 Patentee before: GUANGZHOU XUANWU WIRELESS TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |