CN113177021A - Data export method and device for different data sources - Google Patents
Data export method and device for different data sources Download PDFInfo
- Publication number
- CN113177021A CN113177021A CN202110465823.1A CN202110465823A CN113177021A CN 113177021 A CN113177021 A CN 113177021A CN 202110465823 A CN202110465823 A CN 202110465823A CN 113177021 A CN113177021 A CN 113177021A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- template
- snapshot
- export
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000013515 script Methods 0.000 claims abstract description 110
- 238000012545 processing Methods 0.000 claims description 37
- 238000004590 computer program Methods 0.000 claims description 18
- 238000009795 derivation Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 18
- 230000008569 process Effects 0.000 description 10
- 238000012423 maintenance Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000010354 integration Effects 0.000 description 5
- 230000003993 interaction Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000010261 cell growth Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/186—Templates
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data export method and a data export device of different data sources, which can be used in the financial field or other technical fields. The method comprises the following steps: receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier; inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; wherein the data result sets originate from different data sources; inquiring to obtain a corresponding service template according to the service template identification, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; and reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file. The device is used for executing the method. The data export method and the data export device for different data sources improve the data export efficiency.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data export method and device for different data sources.
Background
For the banking industry, the business processing related to the report forms is very frequent, different businesses have different export requirements, different codes for realizing export functions need to be written for different businesses, and the expansibility and maintainability of the codes are poor.
In order to improve the multiplexing and the expansibility of codes, a Business Intelligence analysis (BI) tool appears in the prior art, and the report export of different services can be realized. However, BI tools only support report exports from the same source database, and for applications with complex data source channels, for example: the data comes from different types of databases, distributed file systems and documents, and the export of the data cannot be realized; it is not well supported by computationally intensive (the derived result requires a large amount of computation).
Therefore, how to propose a data export method of multiple data sources, which can realize data export of different data sources and improve data export efficiency is an important problem to be solved in the field.
Disclosure of Invention
For solving the problems in the prior art, embodiments of the present invention provide a method and an apparatus for exporting data from different data sources, which can at least partially solve the problems in the prior art.
In one aspect, the present invention provides a data export method for different data sources, including:
receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier;
inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources;
inquiring to obtain a corresponding service template according to the service template identification, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset;
and reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file.
In another aspect, the present invention provides a data exporting apparatus for different data sources, including:
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a data export request, and the data export request comprises a service logic identifier and a service template identifier;
the acquisition module is used for inquiring the corresponding service script according to the service logic identifier and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources;
the generating module is used for inquiring and obtaining a corresponding service template according to the service template identification and generating a template snapshot in the memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset;
and the export module is used for reading data from the data result set to the template snapshot based on the information set corresponding to the data result set and writing the data read to the template snapshot into an export file.
In another aspect, the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the data derivation method for different data sources according to any of the above embodiments.
In a further aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data derivation method for different data sources according to any of the embodiments described above.
The data export method and device for different data sources provided by the embodiment of the invention can receive a data export request, wherein the data export request comprises a service logic identifier and a service template identifier, a corresponding service script is inquired according to the service logic identifier, a data result set and a corresponding information set are obtained based on the service script, a corresponding service template is inquired according to the service template identifier, a template snapshot is generated in a memory based on the service template and a snapshot generating rule, data are read from the data result set to the template snapshot based on the information set corresponding to the data result set, and the data read from the template snapshot are written into an export file, so that the export of the data of different data sources is realized, and the data export efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic flow chart of a data export method of different data sources according to a first embodiment of the present invention.
Fig. 2 is a flowchart illustrating a data export method for different data sources according to a second embodiment of the present invention.
Fig. 3 is a flowchart illustrating a data export method for different data sources according to a third embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a data export system of different data sources according to a fourth embodiment of the present invention.
Fig. 5 is a flowchart illustrating a data export method for different data sources according to a fifth embodiment of the present invention.
Fig. 6 is a schematic flow chart of exporting data according to a sixth embodiment of the present invention.
Fig. 7 is a schematic structural diagram of a data deriving apparatus for different data sources according to a seventh embodiment of the present invention.
Fig. 8 is a schematic structural diagram of a data deriving apparatus for different data sources according to an eighth embodiment of the present invention.
Fig. 9 is a schematic structural diagram of a data deriving apparatus for different data sources according to a ninth embodiment of the present invention.
Fig. 10 is a schematic physical structure diagram of an electronic device according to a tenth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
The following describes an implementation process of the data export method for different data sources provided by the embodiment of the present invention by taking a server as an execution subject. It can be understood that the execution subject of the data export method of different data sources provided by the embodiment of the invention includes, but is not limited to, a server.
Fig. 1 is a schematic flowchart of a data export method of different data sources according to a first embodiment of the present invention, and as shown in fig. 1, the data export method of different data sources according to the embodiment of the present invention includes:
s101, receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier;
in particular, a client may send a data export request to a server, which receives the data export request. The data export request includes information such as a service logic identifier, a service template identifier, and a service parameter, and is set according to actual needs. The service logic identifiers correspond to the service scripts one by one; the service template identification corresponds to the service template one by one. The client includes but is not limited to desktop, notebook and other devices.
S102, inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources;
specifically, after receiving the data export request, the server may query a corresponding service script according to the service logic identifier, and then obtain a data result set and a corresponding information set based on the service script. The server may execute the service script locally, obtain service data from different data sources, and integrate the service data to obtain the data result set and the corresponding information set. The server can also send the service script to a data processing server, the data processing server executes the service script, obtains service data from different data sources for integration, obtains the data result set and the corresponding information set, and then returns the data result set and the corresponding information set to the server. The service script is preset and is set according to actual service requirements, and the embodiment of the present invention is not limited. The data result sets originate from different data sources.
S103, inquiring to obtain a corresponding service template according to the service template identifier, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset;
specifically, the server may query and obtain a corresponding service template according to the service template identifier, then analyze the service template to obtain service template related information, and then generate a template snapshot in the memory based on the service template related information and a snapshot generation rule. It can be understood that step S102 and step S103 have no precedence relationship, and the precedence of step S102 and step S103 provided in the embodiment of the present invention is not limited.
The service template is preset, and may include a data source associated with the service template, a column name of each column of data, a storage address of the service template, a template file MD5 code for template tamper resistance, an extension mode of template data, a cell type, an extension direction of a data extension cell, a data type, and other information. The cell types include but are not limited to common cells, variable cells, data expansion cells and the like, and are set according to actual needs. The expansion mode of the template data includes but is not limited to locking expansion or independent expansion, and the locking expansion means that the expansion of the cells can lock the translation of the whole row of cells to the expansion direction; independent expansion means that the expansion of the cells only expands the current cells and does not affect cells in other rows and columns. The expansion direction can comprise downward expansion and rightward expansion, and the expansion direction is set according to actual needs, and the embodiment of the invention is not limited. The data type may include a character type, a number type, a space, etc., and is set according to actual needs, which is not limited in the embodiments of the present invention.
The snapshot generating rule is preset and can include a data obtaining mode of a common cell, a data obtaining mode of a variable cell and a data obtaining mode of a data expansion cell. The data of the common cells are directly obtained from the service template and can comprise constants, digital characters or formulas; the data of the variable cell is obtained through variable query, the variable can be defined in the service template, and the service parameter can be obtained from the data export request as the variable. The data acquisition mode of the data expansion cell can be configured as follows: the data template comprises a data template, and expansion modes, expansion directions, data types, data source names and column names are preset in the data template. The column name corresponds to a field name of each column of data of the data file included in the information set corresponding to the data result.
And S104, reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file.
Specifically, the server reads the data in the data result set into the template snapshot based on the information set corresponding to the data result set, and writes the data read into the template snapshot into the export file. When the data in the data result set is read into the template snapshot, partial data can be read every time, so that the memory occupation is reduced, and the memory overflow is avoided.
The data export method for different data sources provided by the embodiment of the invention can receive a data export request, wherein the data export request comprises a service logic identifier and a service template identifier, a corresponding service script is inquired according to the service logic identifier, a data result set and a corresponding information set are obtained based on the service script, a corresponding service template is inquired according to the service template identifier, a template snapshot is generated in a memory based on the service template and a snapshot generating rule, data is read from the data result set to the template snapshot based on the information set corresponding to the data result set, and the data read from the template snapshot is written into an export file, so that the export of the data of different data sources is realized, and the data export efficiency is improved.
On the basis of the foregoing embodiments, further, the querying a corresponding service script according to the service logic identifier, and obtaining a data result set and a corresponding information set based on the service script includes:
executing the service script to acquire service data from N data sources and integrating the service data to acquire the data result set and the corresponding information set; wherein N is a positive integer greater than or equal to 2.
Specifically, the server may execute the service script to acquire service data from N data sources and perform integration to obtain the data result set and the corresponding information set. Wherein N is a positive integer greater than or equal to 2.
For example, N data sources for acquiring the service data are set in the service script, a service field required for acquiring the service data is set, and a service logic used for integrating the service data is set. When the service script is executed, each data source can be accessed through the access interfaces or the access addresses of the N data sources, corresponding service data are obtained from each data source according to the service fields, and then the service data are integrated according to the service logic to obtain a data result set and a corresponding information set. The data result set comprises at least one data file, data in the data file are stored in rows and columns, and each column of data is separated by a separator; the information set corresponding to the data result set may include description information of the data file, where the description information may include a name of a data source corresponding to the data file, a field name of each column of data of the data file, a size of the data file, and a storage address of the data file. The data source can be MySQL, Qracle and other different databases, distributed file systems and documents.
The following describes the process of obtaining a data result set and a corresponding information set by taking the execution process of the service script of the position summary service query as an example.
Firstly, checking service parameters; the service parameters comprise tree identifications, node identifications and trading desk identifications;
secondly, loading Qracle and Mysql data source objects;
thirdly, inquiring a Mysql database according to the number identification and the trading desk identification to obtain a node level;
fourthly, inquiring a Mysql data table according to the node identification to obtain the node data of the investment portfolio tree; the investment portfolio tree node data comprises a node name dictionary and node identifiers;
fifthly, acquiring a node name dictionary value from the Qracle database according to the node name dictionary;
sixthly, inquiring a Mysql data table according to the node identification obtained in the fourth step to obtain all product information hung under the corresponding node;
seventhly, inquiring an Oracle dictionary item table to obtain account type dictionary item mapping;
eighthly, inquiring an Oracle dictionary item table to obtain poor comment type dictionary items;
and step nine, summarizing the data obtained in the step three to the step seven as the service data according to the service logic rule of position summarization, and generating the data file and the description information of the data file. The position summarizing business logic rule is set according to actual needs, and the embodiment of the invention is not limited; the data file may be a CSV file.
Fig. 2 is a schematic flow chart of a data export method of different data sources according to a second embodiment of the present invention, and as shown in fig. 2, on the basis of the foregoing embodiments, further, the executing the service script to obtain service data from N data sources and integrate the service data, and obtaining a data result set and a corresponding information set includes:
s201, acquiring N data sources from the corresponding configuration files according to the configuration file identifiers; setting the configuration file identifier in the service script;
specifically, when executing the service script, the server may obtain a configuration file identifier from the service script, obtain a corresponding configuration file according to the configuration file identifier, and then obtain N data sources from the configuration file. The configuration file is preset and stores N data sources.
S202, acquiring corresponding service data through N data sources according to each service field; setting each service field in the service script;
specifically, the server obtains service fields needing to derive data from the service script, and then queries corresponding service data from the N data sources according to each service field. Wherein, each service field is set in the service script.
S203, integrating the service data corresponding to each service field according to the service logic rule to generate at least one data file and description information of each data file; wherein the service script sets the service logic rule.
Specifically, after obtaining the service data corresponding to each service field, the server may perform integration processing on the service data corresponding to each service field according to a service logic rule to generate at least one data file, and generate description information for each data file, where the at least one data file forms the data result set, and the description information of each data file forms an information set corresponding to the data result set. The service script sets the service logic rule, and the service logic rule is set according to actual needs, which is not limited in the embodiments of the present invention.
Fig. 3 is a schematic flow chart of a data export method of different data sources according to a third embodiment of the present invention, and as shown in fig. 3, on the basis of the foregoing embodiments, further, the description information of the data file includes a field name of each line of data in the data file; accordingly, the reading data from the data result set into the template snapshot based on the information set corresponding to the data result set comprises:
s301, according to the field name of each line of data in the data file and the template snapshot, transversely expanding a data expansion cell in a memory;
specifically, the template snapshot includes a data expansion cell, and in order to save memory, the data expansion cell only retains an initial position when the template snapshot is generated, and the cell expansion is performed from the initial position when data is read. The server may transversely expand the data expansion cell from an initial position according to a field name of each line of data in the data file, so that each field name corresponds to one cell.
S302, reading data from the data file line by line into the data expansion cells, and expanding the data expansion cells downwards until the data file is read.
Specifically, after the server completes the horizontal expansion of the data expansion cells, the server may read data from the data file into the data expansion cells line by line, so that one data is stored in each cell, then expand one line downward, continue to read the next line of data in the data file, then continue to expand one line downward, and so on until all data in the data file are completely read.
For example, a certain data file includes the following:
1，2，3，4
12，22，32，42
13，23，33，43
14，24，34，44
the description information of the data file includes field names of the first column of data: xx1, field name of second column data: xx2, field name of the third column data: xx3, field name of the fourth column data: xx 4.
The server reads the description information of the data file, transversely expands the cells from the initial position of the data expansion cells of the template snapshot, copies the field names of the data of all columns to the corresponding cells, and then downwardly expands a row of cells to enable each data field to correspond to an empty cell. And the server reads the first row of data from the data file, copies the first row of data into the corresponding cell, then expands a row of cells downwards, and so on until 4 rows of data of the data file are read into the template snapshot. The results obtained after reading the above data file to the template snapshot are shown in table 1.
Table 1 template snapshot after reading data file
| xx1 | xx2 | xx3 | xx4 |
| 1 | 2 | 3 | 4 |
| 12 | 22 | 32 | 42 |
| 13 | 23 | 33 | 34 |
| 14 | 24 | 34 | 44 |
On the basis of the foregoing embodiments, further, the querying a corresponding service script according to the service logic identifier, and obtaining a data result set and a corresponding information set based on the service script includes:
sending a data acquisition request to a data processing server, and receiving a data result set and a corresponding information set returned by the data processing server; the data result set and the corresponding information set are obtained by the data processing server executing the service script to acquire service data from N data sources and integrating the service data; the data acquisition request includes the service script.
Specifically, the server may send a data obtaining request to a data processing server, where the data obtaining request includes the service script. The data processing server may obtain the service script from the data obtaining request, then execute the service script, obtain service data from N data sources and integrate the service data to obtain the data result set and the corresponding information set, and then return the data result set and the corresponding information set to the server. The specific process of the data processing server executing the service script to obtain the data result set and the corresponding information set is similar to the specific process of the server executing the service script to obtain the service data from the N data sources and integrating the service data, and the specific process of obtaining the data result set and the corresponding information set is not repeated here.
On the basis of the foregoing embodiments, further, the writing the data read into the template snapshot into the export file includes:
and calling an export interface of the EXCEL file to write the data in the template snapshot into an export file.
Specifically, the export file may be an EXCEL file, and the server writes the data in the template snapshot into the export file at an export interface that will call the EXCEL file. Wherein the data export request may include a file name, and the export file may be named with the file name included in the data export request.
On the basis of the above embodiments, further, the service script is written in a Groovy language.
Specifically, the Groovy language has expansibility and thermal deployment capability, and the business script can be written by adopting the Groovy language.
Fig. 4 is a schematic structural diagram of a data export system of different data sources according to a fourth embodiment of the present invention, and as shown in fig. 4, the data export system of different data sources according to the embodiment of the present invention includes an interaction module 401, a scheduling module 402, a data logic information maintenance module 403, a template information maintenance module 404, a data calculation module 405, a template snapshot processing module 406, and a data export module 407, where:
the interaction module 401 is configured to receive a data export request sent by a client, where the data export request may include a service logic identifier, a service template identifier, a file name, and a service parameter. The service parameters are set according to actual needs, and the embodiment of the present invention is not limited.
The scheduling module 402 is used to generate data export tasks and register the task state of the data export tasks as running.
The data logic information maintenance module 403 is used to maintain the service scripts, and relevant personnel can generate new service scripts, modify the service scripts, and delete the service scripts through the data logic information maintenance module 403. The data logic information maintenance module 403 may query the corresponding service script according to the service logic identifier. The service script can be written through the Groovy language.
The template information maintenance module 404 is configured to maintain information related to the service template. The service template may include a data source name associated with the service template, a storage address of the service template, an MD5 code of the service template file for template tamper resistance, an extension mode of template data, a cell type, an extension direction of a data extension cell, a data type, and other information. The template information maintenance module 404 may query to obtain a corresponding service template according to the service template identifier.
The data calculation module 405 is used to execute the service script. Before executing the service script, the data calculation module 405 may load context environments including a database connection environment, a distributed file system environment, and a library environment required by the program, such as a general API of the application itself, a Json processing-related, a file processing-related, a database processing-related library environment.
The data calculation module 405 executes the service script to acquire and integrate service data from the N data sources, and acquires a data result set and a corresponding information set. The data result set comprises at least one data file, the data file comprises data needing exporting, the data can be stored in the data file in a row and column mode, and adjacent data in the same row of data are divided into characters. The information set corresponding to the data result set includes description information of the data file, and the description information may include a data source name of the data file, a field name of each column of data of the data file, a size of the data file, a storage address of the data file, and the like.
The template snapshot processing module 406 is configured to generate a template snapshot in the memory based on the service template and the snapshot generation rule. The template snapshot may include normal cells, variable cells, and data expansion cells. The data of the common cells are directly obtained from the service template and can comprise constants, digital characters or formulas; the data of the variable cell is obtained through variable query, and the variable can be defined in the service template or obtained from the data export request. The data acquisition mode of the data expansion cell can be configured as follows: the data template comprises a data template, a data type, an expansion mode, an expansion direction, a data type, a data source name, a column name 1, a column name 2 … … and column names, wherein the expansion mode, the expansion direction, the data type, the data source name and the column names can be obtained by analyzing the data template. The service template may be an EXCEL template file.
The template snapshot processing module 406 analyzes the service template, generates a template snapshot for the configured cells in the service template in the memory, and relocates the positions of the cells of the template snapshot according to the original positions and the data size while specifying the types of the different types of cells.
For the common cells and the variable cells, recording positions of the common cells and the variable cells in the snapshot as predicted positions after the expansion cells are filled; and for the data expansion cell, the position of the data expansion cell is the data expansion starting position after the pattern calculation. Only the template snapshot is stored in the memory, and under the condition of known data source data volume, the position of the final cell is easy to predict, so that the occupation of the memory is reduced, and the function of releasing the format operation pressure in the exporting process in advance is realized.
The data export module 407 is configured to read data from the data result set to the template snapshot based on the information set corresponding to the data result set, and write the data read to the template snapshot to an export file. If the business template is an EXCEL template file, then the export file is an EXCEL file.
Fig. 5 is a schematic flowchart of a data export method of different data sources according to a fifth embodiment of the present invention, and as shown in fig. 5, a specific flow of the data export method of different data sources applied to the data export system of different data sources shown in fig. 4 is as follows:
first, a data export request is received. The interaction module 401 receives a data export request sent by a client, where the data export request may include a service logic identifier, a service template identifier, a file name, and a service parameter.
And secondly, generating a data export task. The scheduling module 402 generates a data export task and registers the task state of the data export task as running.
And thirdly, sending the first query request. The scheduling module 402 sends a first query request to the data logic information maintenance module 403, where the first query request includes a service logic identifier.
And fourthly, returning the service script. The data logic information maintenance module 403 queries the corresponding service script according to the service logic identifier, and returns the service script to the scheduling module 402.
And step five, sending a data acquisition request. The scheduling module 402 sends a data acquisition request to the data calculation module 405, the data acquisition request including the service script.
And sixthly, executing the service script. The data calculation module 405 executes the service script to acquire and integrate service data from the N data sources, and acquires a data result set and a corresponding information set.
And step seven, returning result information. The data calculation module 405 returns the data result set and the corresponding information set to the scheduling module 402.
And step eight, sending a second query request. The scheduling module 402 sends a second query request to the template snapshot processing module 406, the second query request including the service template identification.
And ninthly, generating a template snapshot. The template snapshot processing module 406 is configured to generate a template snapshot in the memory based on the service template and the snapshot generation rule.
Step ten, returning the template snapshot. The template snapshot processing module 406 returns the template snapshot to the scheduling module 402.
And step eleven, exporting data. The scheduling module 402 sends the file name, the service parameter, the template snapshot, the data result set and the corresponding information set to the data export module 407, the data export module 407 reads data from the data result set to the template snapshot based on the information set corresponding to the data result set, obtains data according to the service parameter and writes the data into the template snapshot, and then writes the data read into the template snapshot into an export file, where the file name is the file name of the export file.
Fig. 6 is a schematic flowchart of exporting data according to a sixth embodiment of the present invention, and as shown in fig. 6, a specific flow of exporting data by the data exporting module 407 is as follows:
the method comprises the following steps of firstly, acquiring a data file. The data export module 407 obtains the storage address of the data file from the information set corresponding to the data result set, and obtains the data file according to the storage address of the data file.
And secondly, writing variable cell data. The data export module 407 obtains corresponding data according to the service parameters and writes the data into the variable cell of the template snapshot. The service parameters are set according to actual needs, and the embodiment of the present invention is not limited.
And thirdly, writing data of a row of data expansion cells. The data export module 407 reads a line of data from the data file, and writes the line of data into the data expansion cell of the template snapshot.
And fourthly, adding a row of data expansion cells. The data export module 407 adds a row of data expansion cells to the template snapshot.
And fifthly, judging whether the data file is read completely. The data export module 407 reads the next line of data from the data file, and if the next line of data can be read, the third step is returned to; if the next line of data is not read, the sixth step is entered.
And sixthly, writing the data into the export file. Data export module 407 writes the data in the template snapshot to an export file.
The above process completes the data export of a single data file, and if there are multiple data files, the above process is repeated.
The data export method of different data sources provided by the embodiment of the invention effectively solves the problem of flexible export configuration in an analytical system, solves the problems that the existing system can not support data source media of various platforms, is difficult to realize complex logic export and occupies higher resources during export, effectively improves the agile iteration capability of application development, reduces the development workload, and effectively adapts to the increasing business requirements and complexity.
Fig. 7 is a schematic structural diagram of a data export apparatus of different data sources according to a seventh embodiment of the present invention, and as shown in fig. 7, the data export apparatus of different data sources according to the embodiment of the present invention includes a receiving module 701, an obtaining module 702, a generating module 703 and an exporting module 704, where:
the receiving module 701 is configured to receive a data export request, where the data export request includes a service logic identifier and a service template identifier; the obtaining module 702 is configured to query a corresponding service script according to the service logic identifier, and obtain a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources; the generating module 703 is configured to query and obtain a corresponding service template according to the service template identifier, and generate a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset; the export module 704 is configured to read data from the data result set to the template snapshot based on the information set corresponding to the data result set, and write the data read to the template snapshot to an export file.
Specifically, the client may send a data export request to the receiving module 701, and the receiving module 701 may receive the data export request. The data export request includes information such as a service logic identifier, a service template identifier, and a service parameter, and is set according to actual needs. The service logic identifiers correspond to the service scripts one by one; the service template identification corresponds to the service template one by one. The client includes but is not limited to desktop, notebook and other devices.
After receiving the data export request, the obtaining module 702 may query a corresponding service script according to the service logic identifier, and then obtain a data result set and a corresponding information set based on the service script. The obtaining module 702 may execute the service script locally, obtain service data from different data sources, and integrate the service data to obtain the data result set and the corresponding information set. The obtaining module 702 may further send the service script to a data processing server, where the data processing server executes the service script, obtains service data from different data sources for integration, obtains the data result set and the corresponding information set, and then returns the data result set and the corresponding information set to the obtaining module 702. The service script is preset and is set according to actual service requirements, and the embodiment of the present invention is not limited. The data result sets originate from different data sources.
The generating module 703 may query and obtain a corresponding service template according to the service template identifier, then analyze the service template to obtain service template related information, and then generate a template snapshot in the memory based on the service template related information and the snapshot generating rule.
The export module 704 reads the data in the data result set into the template snapshot based on the information set corresponding to the data result set, and writes the data read into the template snapshot into the export file. When the data in the data result set is read into the template snapshot, partial data can be read every time, so that the memory occupation is reduced, and the memory overflow is avoided.
The data export device for different data sources provided by the embodiment of the invention can receive a data export request, the data export request comprises a service logic identifier and a service template identifier, a corresponding service script is inquired according to the service logic identifier, a data result set and a corresponding information set are obtained based on the service script, a corresponding service template is inquired according to the service template identifier, a template snapshot is generated in a memory based on the service template and a snapshot generating rule, data are read from the data result set to the template snapshot based on the information set corresponding to the data result set, and the data read from the template snapshot are written into an export file, so that the export of the data of different data sources is realized, and the export efficiency of the data is improved.
On the basis of the foregoing embodiments, further, the obtaining module 702 is specifically configured to:
executing the service script to acquire service data from N data sources and integrating the service data to acquire the data result set and the corresponding information set; wherein N is a positive integer greater than or equal to 2.
Fig. 8 is a schematic structural diagram of a data deriving apparatus for different data sources according to an eighth embodiment of the present invention, and as shown in fig. 8, on the basis of the foregoing embodiments, the obtaining module 702 further includes a first obtaining unit 7021, a second obtaining unit 7022, and a generating unit 7023, where:
the first obtaining unit 7021 is configured to obtain N data sources from a corresponding configuration file according to the configuration file identifier; setting the configuration file identifier in the service script; the second obtaining unit 7022 is configured to obtain, according to each service field, corresponding service data through the N data sources; setting the service field in the service script; the generating unit 7023 is configured to perform integration processing on the service data corresponding to each service field according to the service logic rule, and generate at least one data file and description information of each data file; wherein the service script sets the service logic rule.
Fig. 9 is a schematic structural diagram of a data export apparatus of different data sources according to a ninth embodiment of the present invention, as shown in fig. 9, on the basis of the foregoing embodiments, further, the description information of the data file includes a field name of each column of data in the data file; accordingly, the derivation module 704 includes a first extension unit 7041 and a reading unit 7042, wherein:
the first extension unit 7041 is configured to extend a data extension cell in the memory in the transverse direction according to the field name of each line of data in the data file and the template snapshot; the reading unit 7042 is configured to read data from the data file line by line into the data expansion cells, and expand the data expansion cells downward until the data file is read.
On the basis of the foregoing embodiments, further, the obtaining module 702 is specifically configured to:
sending a data acquisition request to a data processing server, and receiving a data result set and a corresponding information set returned by the data processing server; the data result set and the corresponding information set are obtained by the data processing server executing the service script to acquire service data from N data sources and integrating the service data; the data acquisition request includes the service script.
Based on the foregoing embodiments, further, the deriving module 704 is specifically configured to:
and calling an export interface of the EXCEL file to write the data in the template snapshot into an export file.
On the basis of the above embodiments, further, the service script is written in a Groovy language.
The embodiment of the apparatus provided in the embodiment of the present invention may be specifically configured to execute the processing flows of the above method embodiments, and the functions of the apparatus are not described herein again, and refer to the detailed description of the above method embodiments.
It should be noted that the data export method and apparatus for different data sources provided in the embodiment of the present invention may be used in the financial field, and may also be used in any technical field other than the financial field.
Fig. 10 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 10, the electronic device may include: a processor (processor)1001, a communication Interface (communication Interface)1002, a memory (memory)1003 and a communication bus 1004, wherein the processor 1001, the communication Interface 1002 and the memory 1003 complete communication with each other through the communication bus 1004. Processor 1001 may call logic instructions in memory 1003 to perform the following method: receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier; inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources; inquiring to obtain a corresponding service template according to the service template identification, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset; and reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file.
In addition, the logic instructions in the memory 1003 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments, for example, comprising: receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier; inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources; inquiring to obtain a corresponding service template according to the service template identification, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset; and reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file.
The present embodiment provides a computer-readable storage medium, which stores a computer program, where the computer program causes the computer to execute the method provided by the above method embodiments, for example, the method includes: receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier; inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources; inquiring to obtain a corresponding service template according to the service template identification, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset; and reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for data derivation from different data sources, comprising:
receiving a data export request, wherein the data export request comprises a service logic identifier and a service template identifier;
inquiring a corresponding service script according to the service logic identifier, and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources;
inquiring to obtain a corresponding service template according to the service template identification, and generating a template snapshot in a memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset;
and reading data from the data result set to the template snapshot based on the information set corresponding to the data result set, and writing the data read to the template snapshot into an export file.
2. The method of claim 1, wherein querying the corresponding service script according to the service logic identifier and obtaining a data result set and a corresponding information set based on the service script comprises:
executing the service script to acquire service data from N data sources and integrating the service data to acquire the data result set and the corresponding information set; wherein N is a positive integer greater than or equal to 2.
3. The method of claim 2, wherein the executing the service script obtains service data from N data sources and integrates the service data, and obtaining a data result set and a corresponding information set comprises:
acquiring N data sources from the corresponding configuration files according to the configuration file identifiers; setting the configuration file identifier in the service script;
acquiring corresponding service data through N data sources according to each service field; setting each service field in the service script;
integrating the service data corresponding to each service field according to the service logic rule to generate at least one data file and description information of each data file; wherein the service script sets the service logic rule.
4. The method according to claim 3, wherein the description information of the data file comprises a field name of each column of data in the data file; accordingly, the reading data from the data result set into the template snapshot based on the information set corresponding to the data result set comprises:
according to the field name of each line of data in the data file and the template snapshot, transversely expanding data expansion cells in a memory;
and reading data from the data file line by line into the data expansion cells, and expanding the data expansion cells downwards until the data file is read.
5. The method of claim 1, wherein querying the corresponding service script according to the service logic identifier and obtaining a data result set and a corresponding information set based on the service script comprises:
sending a data acquisition request to a data processing server, and receiving a data result set and a corresponding information set returned by the data processing server; the data result set and the corresponding information set are obtained by the data processing server executing the service script to acquire service data from N data sources and integrating the service data; the data acquisition request includes the service script.
6. The method of claim 1, wherein writing the data read into the template snapshot into an export file comprises:
and calling an export interface of the EXCEL file to write the data in the template snapshot into an export file.
7. The method according to any one of claims 1 to 6, wherein the service script is written in a Groovy language.
8. A data derivation apparatus for different data sources, comprising:
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a data export request, and the data export request comprises a service logic identifier and a service template identifier;
the acquisition module is used for inquiring the corresponding service script according to the service logic identifier and acquiring a data result set and a corresponding information set based on the service script; the service script is preset, and the data result set is sourced from different data sources;
the generating module is used for inquiring and obtaining a corresponding service template according to the service template identification and generating a template snapshot in the memory based on the service template and a snapshot generating rule; wherein the service template and the snapshot generating rule are preset;
and the export module is used for reading data from the data result set to the template snapshot based on the information set corresponding to the data result set and writing the data read to the template snapshot into an export file.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110465823.1A CN113177021B (en) | 2021-04-28 | 2021-04-28 | Data export method and device for different data sources |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110465823.1A CN113177021B (en) | 2021-04-28 | 2021-04-28 | Data export method and device for different data sources |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113177021A true CN113177021A (en) | 2021-07-27 |
| CN113177021B CN113177021B (en) | 2024-05-10 |
Family
ID=76926844
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110465823.1A Active CN113177021B (en) | 2021-04-28 | 2021-04-28 | Data export method and device for different data sources |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113177021B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114492376A (en) * | 2021-12-27 | 2022-05-13 | 奇安信科技集团股份有限公司 | Application fingerprint detection method and device and electronic equipment |
| CN117454864A (en) * | 2023-10-26 | 2024-01-26 | 中国人寿保险股份有限公司江苏省分公司 | Display method based on template and data fusion |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080235041A1 (en) * | 2007-03-21 | 2008-09-25 | Cashdollar Jeffrey J | Enterprise data management |
| CN105426465A (en) * | 2015-11-13 | 2016-03-23 | 中国建设银行股份有限公司 | Method and device for online exporting statements |
| CN107153636A (en) * | 2017-04-18 | 2017-09-12 | 北京思特奇信息技术股份有限公司 | It is a kind of to realize the method and system that business datum is quickly exported with PDF format |
| CN108052676A (en) * | 2017-12-28 | 2018-05-18 | 广东金赋科技股份有限公司 | The collocation method and device of multi-data source data |
| CN109766124A (en) * | 2018-12-26 | 2019-05-17 | 深圳左邻永佳科技有限公司 | Business development method, apparatus, computer equipment and storage medium |
| WO2020155760A1 (en) * | 2019-01-28 | 2020-08-06 | 平安科技（深圳）有限公司 | Multi-database data processing method, apparatus, computer device, and storage medium |
-
2021
- 2021-04-28 CN CN202110465823.1A patent/CN113177021B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080235041A1 (en) * | 2007-03-21 | 2008-09-25 | Cashdollar Jeffrey J | Enterprise data management |
| CN105426465A (en) * | 2015-11-13 | 2016-03-23 | 中国建设银行股份有限公司 | Method and device for online exporting statements |
| CN107153636A (en) * | 2017-04-18 | 2017-09-12 | 北京思特奇信息技术股份有限公司 | It is a kind of to realize the method and system that business datum is quickly exported with PDF format |
| CN108052676A (en) * | 2017-12-28 | 2018-05-18 | 广东金赋科技股份有限公司 | The collocation method and device of multi-data source data |
| CN109766124A (en) * | 2018-12-26 | 2019-05-17 | 深圳左邻永佳科技有限公司 | Business development method, apparatus, computer equipment and storage medium |
| WO2020155760A1 (en) * | 2019-01-28 | 2020-08-06 | 平安科技（深圳）有限公司 | Multi-database data processing method, apparatus, computer device, and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 王光增等: "电力企业通用报表综合管理系统设计与实现", 浙江大学学报(工学版), no. 11, 15 November 2009 (2009-11-15) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114492376A (en) * | 2021-12-27 | 2022-05-13 | 奇安信科技集团股份有限公司 | Application fingerprint detection method and device and electronic equipment |
| CN117454864A (en) * | 2023-10-26 | 2024-01-26 | 中国人寿保险股份有限公司江苏省分公司 | Display method based on template and data fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113177021B (en) | 2024-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111367886B (en) | Method and device for data migration in database | |
| CN109670053B (en) | Data object mapping method, device, equipment and computer readable storage medium | |
| US8935575B2 (en) | Test data generation | |
| JP5298117B2 (en) | Data merging in distributed computing | |
| CN113177021B (en) | Data export method and device for different data sources | |
| CN106951231B (en) | Computer software development method and device | |
| CN106557307B (en) | Service data processing method and system | |
| CN113835692A (en) | Dictionary data processing method and device, electronic equipment and computer storage medium | |
| CN115794202A (en) | Data configuration method and engine, file system and computer storage medium | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN114327374A (en) | Business process generation method and device and computer equipment | |
| CN108241620B (en) | Query script generation method and device | |
| CN112860412A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN114116664B (en) | Database table-building sentence processing method, device, computer equipment and storage medium | |
| CN113868138A (en) | Method, system, equipment and storage medium for acquiring test data | |
| CN114168122A (en) | Data script generation method and device | |
| CN109977317B (en) | Data query method and device | |
| CN113326063A (en) | Data processing method, electronic device and storage medium | |
| CN111782608A (en) | Automatic file generation method and device, electronic equipment and storage medium | |
| CN113553329B (en) | Data integration system and method | |
| CN113360494B (en) | Wide-table data generation method, updating method and related device | |
| CN112364007B (en) | Mass data exchange method, device, equipment and storage medium based on database | |
| CN112711437A (en) | Configuration file processing method and device | |
| CN106990997B (en) | Service calling method and device | |
| CN116302524A (en) | Data processing system, method, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |