CN116028454A - Multi-format data file receiving method - Google Patents
Multi-format data file receiving method Download PDFInfo
- Publication number
- CN116028454A CN116028454A CN202310015763.2A CN202310015763A CN116028454A CN 116028454 A CN116028454 A CN 116028454A CN 202310015763 A CN202310015763 A CN 202310015763A CN 116028454 A CN116028454 A CN 116028454A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- file
- model
- turning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a multi-format data file receiving method, which is characterized in that data analysis and data verification are independently processed, and different resolvers are called through the suffix of a reported data file, so that the data reporting supporting multiple formats is realized; automatically reading a defined model file, establishing metadata description of data, reading file data into a memory by a data analyzer through the metadata description, checking data by a data checker through the metadata description, and calling a configured business processing service interface after checking; the meta description of the data supports simple type detection, length detection, regular detection, non-empty detection and the like, a row checker can be configured to perform specified business check, and a model is configured with a data processing service so as to perform checked processing; the data processing supports multi-process load, and the service adopts a multi-thread technology to concurrently execute the report data to be processed. The data receiving and the data processing can be independently deployed, and can also be deployed in a centralized way, so that the multi-machine load is convenient.
Description
Technical Field
The invention relates to the technical field of data receiving, in particular to a multi-format data file receiving method.
Background
The receiving of the reported data is a necessary function of most government administration systems, and the daily administration requirement of the upper government is met by receiving the data of enterprises and lower government. Along with the gradual deepening of informatization, the quantity of reported data is larger and larger at present, a general data receiving system can issue an external interface instruction, so that the data reporting system can report the data into a format required by the interface instruction, and when the reported monitoring systems are more and the format requirements of different monitoring systems on the data are inconsistent, a certain cost is added to an enterprise.
The current market approximation scheme has one of:
scheme one:
and processing the multi-format scientific data file, detecting the format of the data file, calling corresponding analysis components to analyze the data according to the format information, and converting the data through the field mapping relation to achieve the effect of supporting the analysis of the multi-format data file.
The technical disadvantages of the existing scheme are as follows:
1) The existing scheme is biased to reading of data, and lacks support in data verification and service data persistence storage;
2) The data receiving and the data processing are not separated, and the high concurrency scene can not be dealt with;
3) And the related scheme lacking service processing is only biased to read data and store the data in a unified format, and the data cannot be converted according to the meta description of the data.
Disclosure of Invention
In view of the above, in order to solve the above-mentioned problems in the prior art, the present invention proposes a multi-format data file receiving method, which establishes a meta description of data by defining a model file of the data, and a data parser reads the data into a memory through the meta description, and simultaneously completes simple mapping of the data; the data checker checks data through meta description of the data, can perform type detection, length detection, regular detection, non-empty detection and the like, can call a service interface through a configured row check interface, performs necessary service processing on the data, can facilitate enterprises to report the data, and can improve concurrency of a data receiving system.
The invention solves the problems by the following technical means:
a multi-format data file receiving method, comprising the steps of:
dividing the multi-format data file receiving method into a data receiving service and a data processing service, and comprising three basic component services of a data model, data analysis and data verification;
after the data receiving service obtains the data, an adapted data model is obtained through a business class code submitted by an uploading user, an adapted data analyzer is obtained through a file type, the data is simply checked, the data file is persisted on a local disk, and meanwhile, the description information is persisted in a database;
the data processing service reads the uploading record to be processed, acquires the adaptive data model and the data analyzer through the file description information, reads the data into the memory, and calls the data checker to check the data, including basic check and service check;
the data processing service can call the corresponding business processing service in the data model to treat the data after the data analysis and verification are passed.
Preferably, the data model is a meta description of data, so that basic service type and data volume verification can be performed when the data is received.
Preferably, the data model loading process uses xml format to store the appointed data model, and stores the data model in the model directory of the function module resource package.
Preferably, the data model specifically performs the following steps:
s21, searching a data model definition file in an operation package;
s22, judging whether a file to be processed exists, if yes, turning to a step S23, otherwise, turning to a step S25;
s23, reading a data model definition file, converting the data model definition file into a data model object, and storing the data model object into a data model set according to the service class;
s24, turning to a step S22;
s25, finishing the initialization of the data model, and ending.
Preferably, the data model comprises a service type, a display name, a data entity, a row verifier, a service object converter, a maximum allowable number of rows, a service processing service and a data column meta-description; the data column element description comprises a column name, a column display name, whether null is allowed, a data type, a data length and a regular check template; and guiding the data parser and the data verifier to work through the data model.
Preferably, the data receiving service includes the steps of:
when receiving data, a reporter is required to submit service class codes, file types and data file streams according to interface document requirements, a data receiving service obtains a data model through the service class codes, obtains a data analyzer through the file types, and then obtains header information of the data files, wherein the data receiving service comprises the following steps:
1. excel is the header name of the first row;
2. XML and JSON are the names of data items of the first row of data;
after the header information is obtained, comparing the header information with the data column element description in the data model, checking if the header information is consistent, and prompting a reporter to check the service class if the header information is inconsistent; after the header verification is passed, the data line number is obtained, and compared with the maximum allowable record number in the data model, if the data line number is smaller than or equal to the maximum allowable record number, the verification is passed, and if the data line number is larger than the maximum allowable record number, corresponding error information is returned; after the data line number passes the verification, the system stores the data file to the disk and stores the description information of the data file to the database, so that the data receiving is completed.
Preferably, the data processing service includes the steps of:
s1, acquiring a data model according to data file description;
s2, acquiring a data analyzer according to the description of the data file;
s3, the data analyzer analyzes the content of the data file to obtain a data set to be processed;
s4, judging whether all data sets are processed, and if so, turning to step S12; step S5 is carried out if the step is not completed;
s5, taking out a data line to be processed; data verification is carried out on the data items and error information is collected;
s6, if error information exists in the data line processing, the step S4 is carried out; step S7, if not, turning to the step;
s7, judging whether the data model is provided with an object converter, and if so, turning to a step S8; if not, turning to the step S9;
s8, converting the data line by using the object converter to obtain a service entity, and turning to the step S10;
s9, directly converting the service entity through the field name information, and converting to the step S10;
s10, calling a row checker to check a service entity, checking failure, and turning to step S4; checking passing through step S11;
s11, putting the business entity into an entity set, and turning to step S4;
s12, after the data set is processed, judging whether error information exists, if so, turning to a step S13, and if not, turning to a step S14;
s13, storing the error information into the description information of the reported data file, and turning to the step S15;
s14, calling a service processing service to process a service entity set, and turning to the step S15;
s15, ending the processing.
Compared with the prior art, the invention has the beneficial effects that at least:
1) Supporting verification of data and field mapping of service level;
2) The data receiving (acquiring) and the data processing are separated, and the high concurrency scene can be flexibly dealt with in a service load mode;
3) Supporting the business processing of the data and completing the conversion from the original reported data to the business persistence data;
4) The description information of the reported data is stored in a lasting mode, business data corresponding to the data file can be tracked, and problem tracing is facilitated;
5) The data receiving (acquiring) and the data processing are independent, can be deployed in a centralized way, can also be deployed independently, support multi-service load, improve the throughput of the system and cope with high concurrency scenes;
6) And one data model can be configured with one data processing service, and after the data passes the verification, the data processing service is automatically called to finish the lasting storage of business data, so that the data can be conveniently reported by enterprises, and the concurrency of a data receiving system can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method of receiving a multi-format data file according to the present invention;
FIG. 2 is a flow chart of the data model reading of the present invention;
FIG. 3 is a data receiving flow chart of the multi-format data file receiving method of the present invention;
FIG. 4 is a flow chart of data processing of the multi-format data file receiving method of the present invention.
Detailed Description
In order to make the above objects, features and advantages of the present invention more comprehensible, the following detailed description of the technical solution of the present invention refers to the accompanying drawings and specific embodiments. It should be noted that the described embodiments are only some embodiments of the present invention, and not all embodiments, and that all other embodiments obtained by persons skilled in the art without making creative efforts based on the embodiments in the present invention are within the protection scope of the present invention.
The multi-format data file receiving method comprises a data receiving service and a data processing service, wherein the data receiving service obtains a data file through a data receiving interface, an uploading user needs to inform a file type and a service type code, an adaptive data analyzer is called through the file type, an adaptive data model is obtained through the service type code, the header and the data volume information of data are simply checked, the file is stored in a magnetic disk, and description information is stored in a database in a lasting mode; the data processing service acquires the record to be processed, and through the description information of the data file, the data processing service calls the adaptive analyzer to read the data into the memory, then calls the verifier to complete the related verification work, and automatically calls the business processing service.
Referring to fig. 1, the following specifically describes the basic content of the technical solution of the present invention, and the multi-format data file receiving method of the present invention includes the following steps:
1) The system comprises a data receiving service, a data processing service, and a component service with three bases including a data model, data analysis and data verification;
2) After the data receiving service obtains the data, an adapted data model is obtained through a business category code submitted by an uploading user, an adapted data analyzer is obtained through a file type, the data is simply checked, the data file is persisted on a local disk, and meanwhile, the description information is persisted in a database;
3) The data processing service reads the uploading record to be processed, acquires the adaptive data model and the data analyzer through the file description information, reads the data into the memory, and calls the data checker to check the data, wherein the data includes basic check sum service check;
4) The data processing service calls the corresponding business processing service in the data model to treat the data after the data analysis and verification pass.
The multi-format data file receiving method mainly comprises two functions, namely data receiving and data processing. The data model is meta description of data, so that basic service types and data quantity verification can be carried out when the data is received; during data processing, the data line can be correctly acquired, and relevant service verification is performed.
As can be seen from FIG. 2, the data model loading flow of the present invention, the contracted data model is stored in xml format and stored under the model directory of the function module resource package. The specific implementation steps are as follows:
s21, searching a data model definition file in an operation package;
s22, judging whether a file to be processed exists, if yes, turning to a step S23, otherwise, turning to a step S25;
s23, reading a data model definition file, converting the data model definition file into a data model object, and storing the data model object into a data model set according to the service class;
s24, turning to a step S22;
s25, finishing the initialization of the data model, and ending.
The data model comprises service types, display names, data entities, a row verifier, a service object converter, a maximum allowed row number, service processing services and data column element description set information; the data column meta-description contains a column name, a column display name, whether null is allowed, a data type, a data length, a regular check template. And guiding the data analyzer and the data verifier to work through the data model.
As shown in fig. 3, when receiving data, the multi-format data file receiving method requires a reporter to submit a service class code, a file type and a data file stream according to the interface document requirement, the data receiving service obtains a data model through the service class code, obtains a data parser through the file type, and then obtains header information of the data file, wherein:
1. excel is the header name of the first row;
2. XML and JSON are the names of data items of the first row of data;
after the header information is obtained, comparing the header information with the data column element description in the data model, checking if the header information is consistent, and prompting a reporter to check the service class if the header information is inconsistent; after the header verification is passed, the data line number is obtained, and compared with the maximum allowable record number in the data model, if the data line number is smaller than or equal to the maximum allowable record number, the verification is passed, and if the data line number is larger than the maximum allowable record number, corresponding error information is returned; after the data line number passes the verification, the system stores the data file to the disk and stores the description information of the data file to the database, so that the data receiving is completed.
As shown in fig. 4, the data processing flow of the multi-format data file receiving method is divided into three links of data analysis, data verification and service processing, and the figure only shows the processing of a single task, and the specific steps are as follows:
s1, acquiring a data model according to data file description;
s2, acquiring a data analyzer according to the description of the data file;
s3, the data analyzer analyzes the content of the data file to obtain a data set to be processed;
s4, judging whether all data sets are processed, and if so, turning to step S12; step S5 is carried out if the step is not completed;
s5, taking out a data line to be processed; data verification is carried out on the data items and error information is collected;
s6, if error information exists in the data line processing, the step S4 is carried out; step S7, if not, turning to the step;
s7, judging whether the data model is provided with an object converter, and if so, turning to a step S8; if not, turning to the step S9;
s8, converting the data line by using the object converter to obtain a service entity, and turning to the step S10;
s9, directly converting the service entity through the field name information, and converting to the step S10;
s10, calling a row checker to check a service entity, checking failure, and turning to step S4; checking passing through step S11;
s11, putting the business entity into an entity set, and turning to step S4;
s12, after the data set is processed, judging whether error information exists, if so, turning to a step S13, and if not, turning to a step S14;
s13, storing the error information into the description information of the reported data file, and turning to the step S15;
s14, calling a service processing service to process a service entity set, and turning to the step S15;
s15, ending the processing.
According to the multi-format data file receiving method, data receiving and data processing are separated, and only simple verification is carried out during data receiving, so that the concurrent processing capacity of a system is improved; during data processing, the data files to be processed can be rapidly processed by utilizing the multi-thread and multi-process technology to the maximum extent. The invention separates data reading and data checking during data processing, selects a proper data analyzer to read data according to the description information of the data file, then invokes a unified data checker to perform multiple checking on the data under the guidance of a data model, the invention focuses more on the flexibility of data checking and data conversion, the model file supports simple checking of data items, supports entity conversion and supports row checking,
the approximation scheme in the background art focuses on reading data, which is essentially different from the scheme.
The multi-format data file receiving method independently processes data analysis and data verification, and calls different resolvers through the suffix of the reported data file to realize the data reporting supporting multiple formats; after the method is started, a defined model file is automatically read, meta description of data is established, a data analyzer reads file data into a memory through the meta description, a data checker checks the data through the meta description of the data, and a configured service processing interface is called after the data is checked; the meta description of the data supports simple type detection, length detection, regular detection, non-empty detection and the like, a row checker can be configured to perform specified business check, and a model is configured with a data processing service so as to perform checked processing; the description information of the report data is subjected to persistent storage, so that corresponding report files and business data can be tracked, problem tracing is facilitated, and the current processing state of the report files can be checked; the data receiving and the data processing are performed asynchronously, and the data receiving process can call the analyzer to perform simple header and data line number verification. The data processing supports multi-process load, and the service adopts a multi-thread technology to concurrently execute the report data to be processed. The data receiving and the data processing can be independently deployed, and can also be deployed in a centralized way, so that the multi-machine load is convenient.
Aiming at the defects of the background technology, the invention has independent data receiving (acquiring) and data processing, can be deployed in a centralized way, can also be deployed independently, supports multi-service load, improves the throughput of the system and deals with high concurrency scenes; establishing meta description of the data by defining a model file of the data, reading the data into a memory by a data analyzer through the meta description, and simultaneously completing simple mapping of the data; the data checker checks data through meta description of the data, can perform type detection, length detection, regular detection, non-empty detection and the like, and can call a service interface through a configured row check interface to perform necessary service processing on the data; a data model can be configured with a data processing service, and after data passes verification, the data processing service is automatically invoked to complete the lasting storage of business data. Compared with the prior art, the invention has the advantages that:
1) Supporting verification of data and field mapping of service level;
2) The data receiving (acquiring) and the data processing are separated, and the high concurrency scene can be flexibly dealt with in a service load mode;
3) Supporting the business processing of the data and completing the conversion from the original reported data to the business persistence data;
4) And the description information of the reported data is stored in a lasting mode, so that service data corresponding to the data file can be tracked, and problem tracing is facilitated.
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (7)
1. A multi-format data file receiving method, comprising the steps of:
dividing the multi-format data file receiving method into a data receiving service and a data processing service, and comprising three basic component services of a data model, data analysis and data verification;
after the data receiving service obtains the data, an adapted data model is obtained through a business class code submitted by an uploading user, an adapted data analyzer is obtained through a file type, the data is simply checked, the data file is persisted on a local disk, and meanwhile, the description information is persisted in a database;
the data processing service reads the uploading record to be processed, acquires the adaptive data model and the data analyzer through the file description information, reads the data into the memory, and calls the data checker to check the data, including basic check and service check;
the data processing service can call the corresponding business processing service in the data model to treat the data after the data analysis and verification are passed.
2. The multi-format data file receiving method according to claim 1, wherein the data model is a meta description of data, so that basic service type and data volume verification can be performed during data receiving.
3. The method for receiving multiple format data files according to claim 1, wherein the data model loading process, the contracted data model is stored in xml format and stored in a model directory of the resource package of the function module.
4. The multi-format data file receiving method according to claim 1, wherein the data model specifically performs the steps of:
s21, searching a data model definition file in an operation package;
s22, judging whether a file to be processed exists, if yes, turning to a step S23, otherwise, turning to a step S25;
s23, reading a data model definition file, converting the data model definition file into a data model object, and storing the data model object into a data model set according to the service class;
s24, turning to a step S22;
s25, finishing the initialization of the data model, and ending.
5. The method of claim 1, wherein the data model comprises a service type, a display name, a data entity, a row verifier, a service object converter, a maximum allowable number of rows, a service processing service, and a data column meta-description; the data column element description comprises a column name, a column display name, whether null is allowed, a data type, a data length and a regular check template; and guiding the data parser and the data verifier to work through the data model.
6. The multi-format data file receiving method according to claim 1, wherein the data receiving service comprises the steps of:
when receiving data, a reporter is required to submit service class codes, file types and data file streams according to interface document requirements, a data receiving service obtains a data model through the service class codes, obtains a data analyzer through the file types, and then obtains header information of the data files, wherein the data receiving service comprises the following steps:
1. excel is the header name of the first row;
2. XML and JSON are the names of data items of the first row of data;
after the header information is obtained, comparing the header information with the data column element description in the data model, checking if the header information is consistent, and prompting a reporter to check the service class if the header information is inconsistent; after the header verification is passed, the data line number is obtained, and compared with the maximum allowable record number in the data model, if the data line number is smaller than or equal to the maximum allowable record number, the verification is passed, and if the data line number is larger than the maximum allowable record number, corresponding error information is returned; after the data line number passes the verification, the system stores the data file to the disk and stores the description information of the data file to the database, so that the data receiving is completed.
7. The multi-format data file receiving method according to claim 1, wherein the data processing service comprises the steps of:
s1, acquiring a data model according to data file description;
s2, acquiring a data analyzer according to the description of the data file;
s3, the data analyzer analyzes the content of the data file to obtain a data set to be processed;
s4, judging whether all data sets are processed, and if so, turning to step S12; step S5 is carried out if the step is not completed;
s5, taking out a data line to be processed; data verification is carried out on the data items and error information is collected;
s6, if error information exists in the data line processing, the step S4 is carried out; step S7, if not, turning to the step;
s7, judging whether the data model is provided with an object converter, and if so, turning to a step S8; if not, turning to the step S9;
s8, converting the data line by using the object converter to obtain a service entity, and turning to the step S10;
s9, directly converting the service entity through the field name information, and converting to the step S10;
s10, calling a row checker to check a service entity, checking failure, and turning to step S4; checking passing through step S11;
s11, putting the business entity into an entity set, and turning to step S4;
s12, after the data set is processed, judging whether error information exists, if so, turning to a step S13, and if not, turning to a step S14;
s13, storing the error information into the description information of the reported data file, and turning to the step S15;
s14, calling a service processing service to process a service entity set, and turning to the step S15;
s15, ending the processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310015763.2A CN116028454A (en) | 2023-01-06 | 2023-01-06 | Multi-format data file receiving method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310015763.2A CN116028454A (en) | 2023-01-06 | 2023-01-06 | Multi-format data file receiving method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116028454A true CN116028454A (en) | 2023-04-28 |
Family
ID=86073663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310015763.2A Pending CN116028454A (en) | 2023-01-06 | 2023-01-06 | Multi-format data file receiving method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116028454A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117390101A (en) * | 2023-10-16 | 2024-01-12 | 上海易立德信息技术股份有限公司 | Data persistence method and persistence system |
-
2023
- 2023-01-06 CN CN202310015763.2A patent/CN116028454A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117390101A (en) * | 2023-10-16 | 2024-01-12 | 上海易立德信息技术股份有限公司 | Data persistence method and persistence system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7200588B1 (en) | Method and mechanism for analyzing trace data using a database management system | |
| CN102831052B (en) | Test exemple automation generating apparatus and method | |
| CN116028454A (en) | Multi-format data file receiving method | |
| CN112486719A (en) | Method and equipment for RPC interface call failure processing | |
| CN101604290B (en) | Background module test method and backstage module test system | |
| CN104750814B (en) | The automatic storage method of polynary heterogeneous data flow based on multisensor | |
| CN114443468A (en) | Excel table-based test comparison method and system | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| US20220318205A1 (en) | Machine station file processing methods and machine station file processing systems | |
| CN113407565A (en) | Cross-database data query method, device and equipment | |
| CN114154020B (en) | High-capacity data processing method and device based on dynamic label mapping | |
| CN108984757A (en) | A kind of data lead-in method and equipment | |
| CN110457279B (en) | Data offline scanning method and device, server and readable storage medium | |
| CN116680270A (en) | Data table conversion method, device and storage medium | |
| CN113704123B (en) | Interface testing method, device, equipment and storage medium | |
| CN110825795B (en) | Method, apparatus, device and computer readable storage medium for accessing database | |
| CN108304471A (en) | A kind of Heterogeneous data storage method and Heterogeneous data storage device | |
| CN111061749B (en) | Database transaction concurrent redoing method and device based on transaction sequence analysis | |
| CN114371884A (en) | Method, device, equipment and storage medium for processing Flink calculation task | |
| CN114282895A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112597040A (en) | Interface automatic testing method and device and electronic equipment | |
| CN111639936A (en) | Transaction information acquisition method and device, electronic equipment and readable storage medium | |
| CN114363221B (en) | Heartbeat detection method, storage medium, electronic equipment and system based on micro-service system | |
| CN115422126B (en) | Method, system and device for rapidly transferring certificate OFD format file to picture | |
| CN114546942B (en) | Database data reading method, device, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |