CN116821166A - Distributed data export method - Google Patents
Distributed data export method Download PDFInfo
- Publication number
- CN116821166A CN116821166A CN202311111219.4A CN202311111219A CN116821166A CN 116821166 A CN116821166 A CN 116821166A CN 202311111219 A CN202311111219 A CN 202311111219A CN 116821166 A CN116821166 A CN 116821166A
- Authority
- CN
- China
- Prior art keywords
- export
- task
- data
- file
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 18
- 238000004891 communication Methods 0.000 claims abstract description 13
- 238000009877 rendering Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 5
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 239000012634 fragment Substances 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 3
- 238000012885 constant function Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to the technical field of data processing, in particular to a distributed data export method, which comprises the steps of creating a task export template; establishing communication between a data export node and a task scheduling center; the new state is the export task to be executed; the method comprises the steps of broadcasting a data export node cluster of established communication in a slicing way, inquiring an export task which is to be executed and meets export requirements by a data export node, and marking the state as executing; loading service data P of a exporting task and a task exporting template; rendering the export file through the task export template and the service data P to obtain a form file; converting the form file into a document file, and marking the state of the export task as completed; the download status is the document file of the completed export task. The invention is decoupled from the service system, separates a large number of computer IO calculations from the service system, can fully coordinate different computers to complete data export tasks of mass users, and improves the efficiency.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a distributed data export method.
Background
For users of different enterprises, data such as a bill, an order and the like are usually derived in a period of time to assist decision making.
Common ways of deriving distributed data are: 1) For different types of reports, the report is realized by customizing and writing a specific application program; 2) The data export function is coupled with the service system, and other service functions are influenced by the export function; 3) Manually copying the data into a specific report file. In the export of different types of reports, a specific computer application program needs to be customized and written, so that the continuous changing business requirements cannot be rapidly met, and the report has no universality; the data export function is coupled with the service system, when the user quantity is increased, the service system can bear a large amount of data export IO calculation, and the original core service of the service system is influenced; and by means of manual data export, the cost is high, the efficiency is low, and the current application scene is not met.
Accordingly, the present invention provides a method for deriving distributed data, so as to solve at least some of the above technical problems.
Disclosure of Invention
The invention aims to solve the technical problems that: a distributed data export method is provided to solve at least some of the above technical problems.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
a method of distributed data export, comprising the steps of:
step 1, creating different task export templates;
step 2, starting a data export node, and establishing communication between the data export node and a task scheduling center;
step 3, starting data export, wherein the new added state is the export task to be executed in the task database;
step 4, broadcasting the data export node cluster of the established communication in a slicing way, inquiring the export task which is to be executed and meets the export requirement by the data export node, and marking the state as executing;
step 5, loading the business data P of the export task and the corresponding task export template;
step 6, rendering the export file through the task export template and the business data P to obtain a form file;
step 7, converting the form file into a document file, and marking the state of the export task as completed;
and 8, downloading a document file with the completed export task.
Further, the export task comprises a task ID, a task template address, a task data source address, and a task export data type.
Further, the step 4 includes: step 41, the task call center broadcasts the data leading-out node cluster of the established communication in a slicing way according to the set frequency; and 42, after the data export node receives the broadcast, inquiring the task database in batches that the state is to be executed and the value of the task ID modulo the total number of the current data export node cluster is the export task of the current data export node fragment number, and marking the state of the current export task as executing.
Further, the step 5 includes: in the data export node, the data export module loads the service data P through the task data source address of the export task and loads the corresponding task export template through the task template address of the export task.
Further, the export file rendering includes: step 61, in the data export node, an export file is established, the data rendering module analyzes the task export template row by row, if the export file is a non-functional expression, the export file is kept as it is, if the export file is a functional expression, a variable name K is obtained through the functional expression, service data V corresponding to the variable name K is obtained from service data P, and the functional expression is replaced by the service data V; step 62, for the processing of dynamic row and column, variable name K1 is obtained through the function expression, the data set corresponding to variable name K1 is obtained from service data P, the data set is traversed, elements in the data set are added to the export file row by row or column by column, and the style of the elements is kept as the style set by the current function expression, so as to generate a table file.
Further, the step 7 includes: in the data export node, a data file type conversion module converts the form file into a file according to the task export data type in the export task, and uploads the file to a file storage system, and marks the state of the export task as completed.
Further, in the step 8, the task list is exported by the data of the task management page, and the document file whose status is the completed export task is downloaded.
Compared with the prior art, the invention has the following beneficial effects:
according to the invention, the data export nodes which are independently deployed are decoupled from the service system, so that a large number of computer IO (input/output) calculations are separated from the service system, and different computers can be fully coordinated to complete data export tasks of a large number of users; meanwhile, when the data is exported abnormally, the flow of the original core service system is not influenced, and the system stability is improved. The invention can be used for exporting different types of data by only creating a task export template according to actual demands by service personnel and multiplexing the existing service data sources, and can rapidly respond to the continuously changing service demands.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Description of the embodiments
The present invention will be described in further detail with reference to the accompanying drawings, in order to make the objects, technical solutions and advantages of the present invention more apparent. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the method for exporting distributed data provided by the invention comprises the following steps:
step 1, creating different task export templates;
step 2, starting a data export node, and establishing communication between the data export node and a task scheduling center;
step 3, starting data export, wherein the new added state is the export task to be executed in the task database;
step 4, broadcasting the data export node cluster of the established communication in a slicing way, inquiring the export task which is to be executed and meets the export requirement by the data export node, and marking the state as executing;
step 5, loading the business data P of the export task and the corresponding task export template;
step 6, rendering the export file through the task export template and the business data P to obtain a form file;
step 7, converting the form file into a document file, and marking the state of the export task as completed;
and 8, downloading a document file with the completed export task.
In the step 1, before newly adding the export task T to be executed, the user creates a task export template according to the actual service scenario, uploads the export template to the file service system, and obtains the task template address. For the same class of service scenes, the task export templates are universal, and different export tasks of the same class of service scenes can use the same task export templates, except that rendered service data P are different. The exporting task T has a task template address of a corresponding task exporting template to determine a task exporting template used by each exporting task.
In the step 2, the data export node is started, then communication between the data export node and the task scheduling center is established according to the configuration, and the task scheduling center maintains a node list containing all the established communication data export nodes.
In the step 3, the user clicks the data to export, and when the data export is started each time, a new export task with a state to be executed is added to the task database, and the task database contains the export task newly added each time. When creating the export task, the task template address in step 1 is associated. The export task includes, but is not limited to, task basic attributes such as task ID, task template address, task data source address, task export data type, etc.
In the step 4, the task call center broadcasts the data leading-out node cluster of the established communication according to the set frequency (which can be set in a self-defining way); after the data export node receives the broadcast, inquiring the task database in batches that the state is to be executed and the value of the task ID modulo the total number of the current data export node cluster is the export task of the current data export node fragment number, and marking the state of the current export task as executing.
In the step 5, in the data export node, the data export module loads the service data P through the task data source address of the export task and loads the corresponding task export template through the task template address of the export task. The data source address comprises an HTTP address, a Dubbo interface address and the like of a service system, and the acquisition of service data P is completed through standard HTTP request or Dubbo generalized call; and (3) loading a corresponding task export template from the file service system in the step (1) through exporting the task template address of the task. The task derivation template is an Excel template file containing a style, a fixed file and a dynamic change partial function expression F, wherein the function expression F comprises a constant function, a sub-form dynamic row function, a sub-form dynamic column function, a statistical class function and the like, for example, the constant function expression is given by a mode of "$variable name$", the sub-form dynamic row is given by a mode of "$for_row: variable name$", and the digital statistics are given by a mode of "$sum: variable name$". The service data P is JSON formatted data of key and value type containing user derived original data.
In the step 6, in the data export node, an export file is established, the data rendering module analyzes the task export template row by row, if the export file is a non-functional expression, the export file is kept as it is, if the export file is a functional expression, a variable name K is obtained through the functional expression, service data V corresponding to the variable name K is obtained from service data P, and the functional expression on the task export template is replaced by the service data V; for the processing of dynamic row and column, variable name K1 is obtained through a function expression, a data set corresponding to variable name K1 is obtained from service data P, the data set is traversed, elements in the data set are added to a derived file row by row or column by column, the style of the elements is kept to be the style set by the current function expression, and a table file, such as an Excel data file, is generated. The function expression is a placeholder containing a function method name and a service data field name, the placeholder is replaced by actual service data V through different service data P, and a user can see a file with own service data after downloading.
In the step 7, in the data export node, the data file type conversion module converts the form file (xlsx or xls' Excel data file) into a document file corresponding to the export type, such as PDF, word file, etc., according to the task export data type ("PDF", "doc" etc. non-Excel format type) in the export task, and uploads the document file to the file storage system, and marks the status of the export task as completed.
In the step 8, the user exports the task list through the data of the task management page, and downloads the document file with the completed exporting task.
Finally, it should be noted that: the above embodiments are merely preferred embodiments of the present invention for illustrating the technical solution of the present invention, but not limiting the scope of the present invention; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions; that is, even though the main design concept and spirit of the present invention is modified or finished in an insubstantial manner, the technical problem solved by the present invention is still consistent with the present invention, and all the technical problems are included in the protection scope of the present invention; in addition, the technical scheme of the invention is directly or indirectly applied to other related technical fields, and the technical scheme is included in the scope of the invention.
Claims (7)
1. A method of distributed data export, comprising the steps of:
step 1, creating different task export templates;
step 2, starting a data export node, and establishing communication between the data export node and a task scheduling center;
step 3, starting data export, wherein the new added state is the export task to be executed in the task database;
step 4, broadcasting the data export node cluster of the established communication in a slicing way, inquiring the export task which is to be executed and meets the export requirement by the data export node, and marking the state as executing;
step 5, loading the business data P of the export task and the corresponding task export template;
step 6, rendering the export file through the task export template and the business data P to obtain a form file;
step 7, converting the form file into a document file, and marking the state of the export task as completed;
and 8, downloading a document file with the completed export task.
2. A method of distributed data export according to claim 1, wherein the export tasks comprise task IDs, task template addresses, task data source addresses, task export data types.
3. A method of distributed data export according to claim 2, wherein step 4 comprises: step 41, the task call center broadcasts the data leading-out node cluster of the established communication in a slicing way according to the set frequency; and 42, after the data export node receives the broadcast, inquiring the task database in batches that the state is to be executed and the value of the task ID modulo the total number of the current data export node cluster is the export task of the current data export node fragment number, and marking the state of the current export task as executing.
4. A method of distributed data export according to claim 2, wherein step 5 comprises: in the data export node, the data export module loads the service data P through the task data source address of the export task and loads the corresponding task export template through the task template address of the export task.
5. The method of claim 1, wherein the exporting file rendering comprises: step 61, in the data export node, an export file is established, the data rendering module analyzes the task export template row by row, if the export file is a non-functional expression, the export file is kept as it is, if the export file is a functional expression, a variable name K is obtained through the functional expression, service data V corresponding to the variable name K is obtained from service data P, and the functional expression is replaced by the service data V; step 62, for the processing of dynamic row and column, variable name K1 is obtained through the function expression, the data set corresponding to variable name K1 is obtained from service data P, the data set is traversed, elements in the data set are added to the export file row by row or column by column, and the style of the elements is kept as the style set by the current function expression, so as to generate a table file.
6. A method of distributed data export according to claim 2, wherein step 7 comprises: in the data export node, a data file type conversion module converts the form file into a file according to the task export data type in the export task, and uploads the file to a file storage system, and marks the state of the export task as completed.
7. A method of distributed data export according to claim 1, wherein in step 8, the task list is exported via the data of the task management page, and the document file with the status of the completed export task is downloaded.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311111219.4A CN116821166B (en) | 2023-08-31 | 2023-08-31 | Distributed data export method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311111219.4A CN116821166B (en) | 2023-08-31 | 2023-08-31 | Distributed data export method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116821166A true CN116821166A (en) | 2023-09-29 |
| CN116821166B CN116821166B (en) | 2023-12-08 |
Family
ID=88139653
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311111219.4A Active CN116821166B (en) | 2023-08-31 | 2023-08-31 | Distributed data export method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116821166B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100146050A1 (en) * | 2008-12-05 | 2010-06-10 | Amalto Technologies Corp. | Distributed document transformation for electronic business to business transactions |
| CN103412765A (en) * | 2013-09-04 | 2013-11-27 | 广东全通教育股份有限公司 | Method for realizing universality of data exporting interface |
| CN110990465A (en) * | 2019-11-28 | 2020-04-10 | 北京明略软件系统有限公司 | Data export method and device, electronic equipment and storage medium |
| CN112463364A (en) * | 2020-11-09 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Packet-based distributed storage SCSI target service management method and system |
| CN113434278A (en) * | 2021-07-08 | 2021-09-24 | 上海浦东发展银行股份有限公司 | Data aggregation system, method, electronic device, and storage medium |
| CN113901098A (en) * | 2021-12-13 | 2022-01-07 | 北京快成科技有限公司 | Template-based data export method, device, medium and equipment |
| CN114004209A (en) * | 2021-10-29 | 2022-02-01 | 平安国际智慧城市科技股份有限公司 | PDF format data export method and device, electronic equipment and readable storage medium |
| CN114416438A (en) * | 2022-01-25 | 2022-04-29 | 中国工商银行股份有限公司 | Data export method and device, computer equipment and scheduling service system |
-
2023
- 2023-08-31 CN CN202311111219.4A patent/CN116821166B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100146050A1 (en) * | 2008-12-05 | 2010-06-10 | Amalto Technologies Corp. | Distributed document transformation for electronic business to business transactions |
| CN103412765A (en) * | 2013-09-04 | 2013-11-27 | 广东全通教育股份有限公司 | Method for realizing universality of data exporting interface |
| CN110990465A (en) * | 2019-11-28 | 2020-04-10 | 北京明略软件系统有限公司 | Data export method and device, electronic equipment and storage medium |
| CN112463364A (en) * | 2020-11-09 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Packet-based distributed storage SCSI target service management method and system |
| CN113434278A (en) * | 2021-07-08 | 2021-09-24 | 上海浦东发展银行股份有限公司 | Data aggregation system, method, electronic device, and storage medium |
| CN114004209A (en) * | 2021-10-29 | 2022-02-01 | 平安国际智慧城市科技股份有限公司 | PDF format data export method and device, electronic equipment and readable storage medium |
| CN113901098A (en) * | 2021-12-13 | 2022-01-07 | 北京快成科技有限公司 | Template-based data export method, device, medium and equipment |
| CN114416438A (en) * | 2022-01-25 | 2022-04-29 | 中国工商银行股份有限公司 | Data export method and device, computer equipment and scheduling service system |
Non-Patent Citations (3)
| Title |
|---|
| AMBREEN HUSSAIN 等: "Implementation of OGC Compliant Framework for Data Integration in Water Distribution System", PROCEDIA ENGINEERING, vol. 119, pages 1366 - 1374, XP029263463, DOI: 10.1016/j.proeng.2015.08.984 * |
| 牟德志;: "基于Lotus Notes的OA系统中的OLE应用", 信息技术, vol. 33, no. 12, pages 144 - 146 * |
| 黄蔚;张;李军怀;白敏;: "非结构化Word数据表与RDB间的存储转换", 计算机工程, vol. 35, no. 20, pages 37 - 40 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116821166B (en) | 2023-12-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108196961B (en) | Asynchronous message processing method, terminal, system and storage medium | |
| CN109814856B (en) | Data entry method, device, terminal and computer readable storage medium | |
| CN101827302A (en) | Multi-service unified processing method and unified service platform | |
| CN111818175B (en) | Enterprise service bus configuration file generation method, device, equipment and storage medium | |
| CN114722119A (en) | Data synchronization method and system | |
| CN101771930A (en) | User terminal, software hosting business management system and management method | |
| CN112464632A (en) | Form style dynamic storage and conversion method under excel report | |
| CN116821166B (en) | Distributed data export method | |
| US8566279B1 (en) | Enhanced data collection techniques | |
| CN103561113A (en) | Web Service interface generating method and device | |
| WO2022073401A1 (en) | Activiti-based flow chart adjustment method and apparatus, electronic device, and storage medium | |
| CN111930690A (en) | File generation method and device | |
| CN112835573A (en) | Data query method and device, electronic equipment and storage medium | |
| CN112650777A (en) | Data warehouse manufacturing method and device, terminal equipment and computer storage medium | |
| CN113076107A (en) | Method for automatically acquiring and fusing logs through finite state machine | |
| CN112688998B (en) | Main data subscription pushing method capable of configuring with permission | |
| CN112988879A (en) | Method, system, device, storage medium and processor for accessing database | |
| CN100498765C (en) | Method and device for making electric newspaper printing plate | |
| US10348864B2 (en) | Information transfer method, client, and server | |
| CN116702111B (en) | Webpage form data authority control method and computer storage medium | |
| CN117034900B (en) | Automatic generation method of API (application program interface) code file, terminal equipment and storage medium | |
| CN113890872B (en) | Data set uploading method and device, electronic equipment and storage medium | |
| CN111522871B (en) | Order data aggregation method, device and storage medium | |
| CN113468460B (en) | Method, device and equipment for rendering Web page based on YAML (Yaml language) | |
| CN116880820A (en) | Visual generation method and device for application, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |