CN111782718A - Plug-in data reporting system and data reporting method - Google Patents
Plug-in data reporting system and data reporting method Download PDFInfo
- Publication number
- CN111782718A CN111782718A CN202010799238.0A CN202010799238A CN111782718A CN 111782718 A CN111782718 A CN 111782718A CN 202010799238 A CN202010799238 A CN 202010799238A CN 111782718 A CN111782718 A CN 111782718A
- Authority
- CN
- China
- Prior art keywords
- plug
- data
- target
- storage
- verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000003860 storage Methods 0.000 claims abstract description 109
- 238000012795 verification Methods 0.000 claims abstract description 75
- 238000013500 data storage Methods 0.000 claims abstract description 35
- 238000004458 analytical method Methods 0.000 claims abstract description 28
- 238000013524 data verification Methods 0.000 claims abstract description 24
- 238000006243 chemical reaction Methods 0.000 claims abstract description 21
- 238000012384 transportation and delivery Methods 0.000 claims description 18
- 238000007689 inspection Methods 0.000 claims description 3
- 238000013501 data transformation Methods 0.000 claims 1
- 238000007405 data analysis Methods 0.000 abstract 2
- 238000007726 management method Methods 0.000 description 18
- 238000011161 development Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 238000013507 mapping Methods 0.000 description 8
- 230000008520 organization Effects 0.000 description 6
- 238000009826 distribution Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 239000008186 active pharmaceutical agent Substances 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 239000003550 marker Substances 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000001105 regulatory effect Effects 0.000 description 3
- 238000002955 isolation Methods 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000002716 delivery method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
- G06F9/44526—Plug-ins; Add-ons
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a plug-in data reporting system and a data reporting method. The data reporting system comprises a plug-in library. The plug-in library includes a plurality of plug-ins. The data reporting system further comprises a data access component, a data analysis and conversion component, a data verification component and a data storage component. When a user submits data through a data submission system, the data access component, the data analysis and conversion component, the data verification component and the data storage component can call a target plug-in the plug-in library to respectively realize the access, analysis and conversion, verification and storage of the data.
Description
Technical Field
The present application relates to the field of information technology, and in particular, to a plug-in data delivery system and a data delivery method.
Background
In the digital era, the requirements of acquisition, reporting and management of various data are widely existed in the fields of supervision, management, statistical audit and the like. For example, the chinese people bank needs to collect data from the supervised banking institution to perform summary statistics and supervision of financial data. The local financial bureau also needs to collect data from the administered financial institutions to realize the supervision of the financial institutions. Different fields and different scenes have different data formats, different data reporting modes, different data storage modes and different reporting periods. At present, in each field, professional development is carried out through a professional customization mode to realize data submission, the development period is long, and manpower and financial resources are consumed. For example, the conventional data reporting scheme is hard-coded development according to the data format content of the data reported by the user, and code development is required whenever the user needs to add a modified data format, add a data reporting mode or a data storage mode, which is time-consuming, labor-consuming and difficult to maintain.
Disclosure of Invention
For the time human cost who solves traditional customization data newspaper mode and bring high and the difficult technical problem of maintenance, this application discloses a plug-in components ization data newspaper system, includes: the plug-in library comprises a plurality of plug-ins; the data access component is configured to receive a message from a transmitting end, and the message comprises target data; a data parsing conversion component comprising at least one first plug-in interface and configured to: receiving the target data from the data access assembly, selecting a target analysis plug-in from at least one analysis plug-in, sending the target data to the target analysis plug-in through the at least one first plug-in interface to analyze the target data, receiving the analyzed analysis data of the target data from the target analysis plug-in through the at least one first plug-in interface, and converting the format of the analyzed data into a target format; a data verification component configured to verify the parsed data; and a data storage component configured to store the parsed data to a target storage system of a storage medium.
In some embodiments, wherein each of the at least one parsing plug-in comprises: the data input format of the data input interface corresponds to the data format of the output end of the first plug-in interface; and the data output interface outputs data with a format corresponding to that of the input data end of the first plug-in interface.
In some embodiments, wherein the at least one parsing plug-in comprises at least one of an Excel parsing engine, a CSV streaming parsing engine, an API data conversion engine, and an XML/DAT format parsing engine.
In some embodiments, wherein the data verification component comprises at least one second plug-in interface and is configured to: selecting a target verification plug-in from at least one data verification plug-in; sending the analysis data to the target inspection plug-in through the at least one second plug-in interface; and receiving a verification result from the target verification plug-in through the at least one second plug-in interface.
In some embodiments, the target verification plug-in verifies the parsed data according to a preset verification rule, wherein the verification rule includes at least one of a format verification rule, an intra-row rule verification rule, a uniqueness verification rule, a script verification rule, and an inter-table verification rule.
In some embodiments, the at least one second plug-in interface selects the target verification plug-in from the at least one verification plug-in according to configuration information.
In some embodiments, wherein the data storage component comprises at least one third plug-in interface and is configured to: selecting a target storage plug-in from at least one data storage plug-in; sending the verified data to the target storage plug-in through the at least one third plug-in interface; and receiving a storage result from the target storage plug-in through the at least one third plug-in interface.
In some embodiments, wherein the at least one storage plug-in comprises at least one of a MySQL database storage plug-in, an OceanBase database storage plug-in, a HIVE database storage plug-in, an Oracel database storage plug-in, a SQL Server database storage plug-in, a local file system storage plug-in, a network file system NFS storage plug-in, a Hadoop file system HDFS storage plug-in, and an object storage OSS storage plug-in; and the storage result comprises a storage path of the target data.
In some embodiments, the at least one third plug-in interface selects the target storage plug-in from the at least one storage plug-in according to configuration information.
In some embodiments, wherein the data access component comprises at least one fourth plug-in interface and is configured to: selecting a target access plug-in from at least one data access plug-in; sending the message to the target access plug-in through the at least one fourth plug-in interface; and receiving the data of the target data after decryption and integrity verification from the target access plug-in through the at least one fourth plug-in interface.
In some embodiments, the at least one data access plug-in comprises at least one of a manual web page upload plug-in, a pre-processor post-verification automatic upload plug-in, an OpenAPI call upload plug-in, and a web page fill plug-in.
In some embodiments, the data delivery system further comprises: a configuration component comprising at least one configuration interface configured to receive configuration information.
The application also discloses a data reporting method, which comprises the following steps: receiving a message from a transmitting end through at least one data access plug-in, wherein the message comprises target data; loading a target analysis plug-in according to a preset analysis rule, analyzing the target data and converting the target data into a target format; loading a target verification plug-in according to a preset verification rule, and verifying the analyzed target data; and loading the target storage plug-in according to a preset storage rule, and storing the verified target data to the target storage system.
The data submission system can flexibly and extendably support different report template formats, different data formats, different submission modes, different verification modes and different storage modes by adopting layered, plug-in, componentized and configured architecture design, so that the data submission system can be widely reused in various scenes in various fields, and the defects of long development period, labor and financial consumption, difficulty in maintenance and the like caused by customized development are avoided.
Drawings
Fig. 1 is a schematic view illustrating an application scenario of a datagram delivery system according to an embodiment of the present application;
fig. 2 is a schematic diagram illustrating a hardware structure of a datagram system according to an embodiment of the present application; and
fig. 3 shows a block diagram of an architecture of a datagram system according to an embodiment of the present application.
Detailed Description
The following description is presented to enable any person skilled in the art to make and use the present disclosure, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present application. Thus, the present application is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.
The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting.
These and other features of the present application, as well as the operation and function of the related elements of structure and the combination of parts and economies of manufacture, may be significantly improved upon consideration of the following description. All of which form a part of this application, with reference to the accompanying drawings. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the application.
These and other features of the present application, as well as the operation and function of the related elements of the structure, and the economic efficiency of assembly and manufacture, are significantly improved by the following description. All of which form a part of this application with reference to the accompanying drawings. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the application. It should also be understood that the drawings are not drawn to scale.
Fig. 1 is a schematic view illustrating an application scenario of a datagram system 300 according to an embodiment of the present application. Specifically, the application scenario may include the data delivery system 300, the terminal device 130, the network 120, and the database 150.
The user 110 can submit the report file to the data submission system 300 through the terminal 130. The user 110 can submit the report file to the ticketing system 300 through the front desk of the ticketing system 300 installed on the terminal 130. By way of example, the foreground may include, but is not limited to, a browser, an application, and the like.
The manner in which user 110 submits the report file to the data submission system may be varied. By way of example, the reporting mode may include, but is not limited to, manual webpage uploading, automatic uploading after the pre-processor checks, webpage filling, invoking open application programming interface (OpenAPI) reporting integrated in a third-party system, and the like.
The terminal device 130 may comprise a smart device carrying the report. For example, the terminal device may be a terminal device of a branch of the financial institution, or may be a terminal device of a customer 110 served by the financial institution or a regulatory agency of the financial institution. In some embodiments, the terminal device 130 may include a built-in device of the mobile device 130-1, the tablet 130-2, the laptop 130-3, the motor vehicle 130-4, or the like, or any combination thereof. In some embodiments, the terminal device 130 may have one or more applications installed. The application can provide the user 110 with the ability to interact with the outside world, as well as an interface, over the network 120. The target application may be any application running on the terminal device 130. For example, the target application may be an application program with data statistics function of the financial service institution.
The database 150 may store data mapping relationships or instructions that perform the methods of table content mapping described herein. For example, database 150 may be one or more databases of a data center of the financial institution. Database 150 may be a centralized database or a decentralized database.
The messaging system 300 may include one or more servers. For example, the data submission system 300 may be one or more servers of a data center of a financial institution. For convenience of description, in the following description of the present application, the data distribution system 300 of the present application is described by taking a server as an example. As an example, fig. 2 shows a hardware structure diagram of a datagram delivery system 300 provided according to an embodiment of the present application.
As shown in fig. 2, the messaging system 300 includes at least one memory 230 and at least one processor 220. In some embodiments, the messaging system 300 may also include a communication port 250 and an internal communication bus 210. Meanwhile, the data messaging system 300 may also include an I/O component 260.
I/O component 260 supports input/output between the messaging system 300 and other components (e.g., terminal device 130).
The communication port 250 is used for data communication between the messaging system 300 and the outside. For example, the data distribution system 300 may be connected to the network 120 through the communication port 250, and further receive the data table sent by the user 110, and further send the data to the database 150 through the communication port 250 according to the table mapping relationship, and store the content of the data table in the database 150.
The at least one processor 220 communicates with the at least one memory 230 via an internal communication bus 210. The at least one processor 220 is configured to execute the at least one set of instructions, and when the at least one processor 220 executes the at least one set of instructions, the messaging system 300 implements the method for table content mapping provided herein. Processor 220 may perform all the steps involved in the method of table content mapping. Processor 220 may be in the form of one or more processors, and in some embodiments, processor 220 may include one or more hardware processors, such as microcontrollers, microprocessors, Reduced Instruction Set Computers (RISC), Application Specific Integrated Circuits (ASICs), application specific instruction set processors (ASIPs), Central Processing Units (CPUs), Graphics Processing Units (GPUs), Physical Processing Units (PPUs), microcontroller units, Digital Signal Processors (DSPs), Field Programmable Gate Arrays (FPGAs), Advanced RISC Machines (ARM), Programmable Logic Devices (PLDs), any circuit or processor capable of executing one or more functions, or the like, or any combination thereof. For illustrative purposes only, only one processor 220 is depicted in the messaging system 300 in the present application. It should be noted, however, that the messaging system 300 may also include multiple processors and that, thus, the operations and/or method steps disclosed herein may be performed by a single processor, as described herein, or by a combination of multiple processors. For example, if in the present application processor 220 of the messaging system 300 performs steps a and B, it should be understood that steps a and B may also be performed by two different processors 220, either jointly or separately (e.g., a first processor performing step a, a second processor performing step B, or both a first and second processor performing steps a and B).
Fig. 3 shows a block diagram of an architecture of a datagram system 300 according to an embodiment of the present application. In particular, the datagram system 300 may include a data access component 310, a data parsing conversion component 320, a data verification component 330, and a data storage component 340.
The messaging system 300 may include one or more plug-in libraries. The plug-in library may include a number of plug-ins. Each of the number of plug-ins may include one or more sets of instructions.
The data access component 310 may be one or more sets of instructions stored within the memory 230. Data access component 310 may be part of the host application of the messaging system 300.
The data access component 310 is configured to receive a message from a sender. As an example, the sender may be the terminal 130 in the application scenario shown in fig. 1.
The data access component 310 may include at least one fourth plug-in interface. The fourth plug-in interface may be a standard interface provided by data access component 310 as part of the host application of the messaging system 300. The standard interface may allow third parties to write plug-ins to interact with the data access component 310. The third party written plug-ins include programs, routines, objects, components, data structures, procedures, modules, and the like that may be independent of the host application. The host application provides services that the plug-in can use, allowing the plug-in to register the plug-in itself in the host application, as well as protocols for data exchange with the plug-in.
The fourth plug-in interface may include one or more interface documents that are predefined and stored in memory 230. The interface document may define a specification for the data access component 310 to interact with a third party authoring plug-in. By way of example, the interface document may include, but is not limited to, methods, uri, request parameters, return parameters, and the like. The methods may include, but are not limited to, add (post), modify (put), delete (delete), get (get), and the like.
The fourth plug-in interface can be used for plug-in data access plug-in. The data access plug-in may receive data reported by user 110 through terminal 130. In some embodiments, the data access plug-in may also decrypt and integrity verify the received data.
The fourth plugin interface is configured to: selecting a corresponding target access plug-in from at least one data access plug-in; sending the message to the target access plug-in through the at least one fourth plug-in interface; and receiving the data of the target data after decryption and integrity verification from the target access plug-in through the at least one fourth plug-in interface.
The at least one data access plug-in comprises at least one of a manual webpage uploading plug-in, a pre-processor verified automatic uploading plug-in, an OpenAPI call uploading plug-in and a webpage filling plug-in.
The data access component 310 can invoke the target data access plugin through the fourth plugin interface to enable receipt of data.
In some embodiments, the data access component 310 may invoke a target access plug-in corresponding to a pre-configured delivery manner according to a parameter of the delivery manner. For example, if the reporting mode is "web filling" configured by the configuration information in advance, when the user 110 reports the report through the web page displayed on the terminal 130, the data access component 310 automatically calls the target access plug-in corresponding to the "web filling" mode according to the configuration information, and receives the message from the reporting end through the target access plug-in.
In some embodiments, the data access component 310 may automatically invoke the target access plug-in according to the data type of the report submitted by the user 110.
In some embodiments, the data access component 310 may also invoke the target access plug-in according to a parameter identifying a delivery manner carried in a message delivered by a delivery end.
The at least one data access plug-in refers to an instruction that implements the fourth interface specification of the data access component 310. The at least one data access plug-in identifies itself by implementing the second interface and receives event responses from the host program. The data access component 310 identifies the identity of the data access plug-in through the fourth interface.
The data parsing conversion component 320 can be one or more sets of instructions stored within the memory 230. The data parsing component 320 can be part of the host application of the data delivery system 300. The data parsing conversion component 320 can receive the target data from the data access component 310. The data parsing conversion component 320 can parse the target data.
The data parsing conversion component 320 can include at least one first plug-in interface. The first plug-in interface allows data parsing conversion plug-ins that meet their specifications to interact with the data distribution system 300.
The data parsing conversion component 320 can plug-in one or more parsing plug-ins. The parsing plug-in refers to an instruction that implements the first plug-in interface. The parsing plug-in may parse the target data. In some embodiments, the parsing plug-in may convert the format of the target data to a target data format.
The data parsing conversion component 320 is configured to: the target data from the data access assembly is received, a target analysis plug-in is selected from at least one analysis plug-in, the target data is sent to the target analysis plug-in through the at least one first plug-in interface to analyze the target data, the analysis data after the target data is analyzed is received from the target analysis plug-in through the at least one first plug-in interface, and the format of the analysis data is converted into a target format.
The data parsing converting component 320 may select a target parsing plug-in from at least one parsing plug-in, load the target parsing plug-in to parse the target data, and convert a format of the parsed data into a target format.
By way of example, the at least one parsing plug-in includes at least one of an Excel parsing engine, a CSV streaming parsing engine, an API data conversion engine, and an XML/DAT format parsing engine.
Each of the at least one parsing plug-in comprises: the data input format of the data input interface corresponds to the data format of the output end of the first plug-in interface, and the data output format of the data input interface corresponds to the data format of the input data end of the first plug-in interface.
In this way, the parsing plug-in can convert the target data received from the data access component 310 into a target data format.
The target analysis plug-in can analyze the data according to a preset analysis rule. As an example, the parsing rule may include a positioning policy for each data item in the report. For example, in some embodiments, for a data item in a one-dimensional detail type report, the data item may be located based on the column number of the column in which the data item is located. For example, in some embodiments, for a data item located in a fixed cell in a two-dimensional report, the data item may be located based on the coordinates of the cell in which the data item is located. For example, in some embodiments, for a data item in a two-dimensional report that is located in an unfixed cell, the data item may be located based on the paragraph start marker text, the paragraph end marker text, and the left marker text of the data item.
In some embodiments, the data parsing conversion component 320 can select the target parsing plug-in according to the type of data. For example, when the user 110 uploads an Excel report through the terminal 130, the data parsing conversion component 320 automatically selects an Excel parsing engine to parse the data.
In some embodiments, the data parsing conversion component 320 may invoke a target parsing plug-in corresponding to the parsing parameters according to the preconfigured parsing parameters. For example, the monitoring organization configures a certain report parsing manner as EXCEL parsing in the configuration information, and then, when the user 110 reports the report through the terminal 130, the data parsing and converting component 320 may call an EXCEL parsing engine corresponding to the configured parameter according to the configured parameter, and parse the target data through the EXCEL parsing engine.
The data verification component 330 may include at least one second plug-in interface. The second plug-in interface allows a data verification plug-in that meets its specifications to interact with the host application of the data delivery system 300.
The data verification component 330 can plug-in one or more verification plug-ins. The verification plug-in refers to an instruction that implements the second plug-in interface. The verification plug-in may verify the parsed data.
The data verification component 330 may select a target verification plug-in from at least one verification plug-in, and load the target verification plug-in to verify the parsed data.
The data verification component 330 is configured to: selecting a corresponding target verification plug-in from at least one data verification plug-in; sending the analysis data to the target inspection plug-in through the at least one second plug-in interface; and receiving a verification result from the target verification plug-in through the at least one second plug-in interface.
In some embodiments, the data verification component 330 may invoke a target verification plug-in corresponding to the verification parameter according to a preconfigured verification parameter. For example, the monitoring authority configures some report verification modes including format verification and uniqueness verification in the configuration information, so that when the user 110 reports the report through the terminal 130, the data verification component 330 may call a verification plug-in corresponding to the configured parameter according to the configured parameter, and verify the target data through the verification plug-in.
In some embodiments, the target verification plug-in may verify the parsed data according to a preset verification rule. By way of example, the verification rules may include, but are not limited to, format verification rules, inline rule verification rules, uniqueness verification rules, script verification rules, inter-table verification rules, and the like.
By way of example, the data checking rules may specify the format, value range, validity checking rules, etc. of the data items in the report. For example, some of the data items in the report are in text format, and some are in numerical format; then, whether the format of these data items conforms to the preset format. For example, whether the value of the data item is within a preset range of values. For example, for the identity data, the mobile phone number data, and the like, whether the numerical length meets the requirement or not is valid data. In addition, the check rule can also comprise a cross check rule among the data items in the report. For example, whether the data items follow a preset rule, for example, whether the product of the 7 th column data and the 4 th column data is equal to the 6 th column data, and so on.
The verification result of the target verification plug-in on the target data can comprise pass or fail. When the verification result shows that the target data passes the verification, the data verification component 330 sends the data passing the verification to the data storage component 340. When the verification result shows that the target data does not pass the verification, the data research group 330 may notify the user of the change of the data by displaying notification information on a display device of the terminal 130.
The data storage component 340 may be one or more sets of instructions stored within the memory 230. Data storage component 340 may be part of the host application of the datagram system 300. The data storage component 340 may store the parsed and verified data to the target storage system 390.
The storage system 390 may include a structured data storage system and an unstructured data storage system.
By way of example, the structured storage system may include, but is not limited to, a MySQL database, an OceanBase database, a HIVE database, an Oracel database, an SQL Server database, and the like.
By way of example, the unstructured storage system may include, but is not limited to, a local file system, a network file system NFS, a Hadoop file system HDFS, an object store OSS, and so forth.
The data storage component 340 can include at least one third plug-in interface. The third plug-in interface allows data storage plug-ins that meet their specifications to interact with the data delivery system 300.
The data storage component 340 can plug-in one or more storage plug-ins. The memory plug-in refers to an instruction that implements the third plug-in interface. The storage plug-in may store the parsed and verified data to a target storage system.
The data storage component 340 may select a target storage plug-in from the at least one storage plug-in, load the target storage plug-in, and store the parsed and verified data to the target storage system. The data storage component 340 can select a target storage plug-in from the at least one data storage plug-in; sending the verified data to the target storage plug-in through the at least one third plug-in interface; and receiving a storage result from the target storage plug-in through the at least one third plug-in interface.
By way of example, the at least one storage plug-in may include, but is not limited to, a MySQL database storage plug-in, an OceanBase database storage plug-in, a HIVE database storage plug-in, an Oracel database storage plug-in, a SQL Server database storage plug-in, a local file system storage plug-in, a network file system NFS storage plug-in, a Hadoop file system HDFS storage plug-in, an object storage OSS storage plug-in, and so forth.
In some embodiments, the data storage component 340 may invoke the target storage plug-in corresponding to the storage parameter according to a preconfigured storage parameter. For example, if the storage mode is configured to be a file storage system in advance through the configuration information, when the user 110 reports a report through a webpage displayed on the terminal 130, the data storage component 340 may automatically invoke a file system storage plug-in corresponding to the parameter according to the configuration information, and store the parsed and verified data to the target file system through the file system storage plug-in.
The target storage plug-in can store the analyzed and verified data to the target storage system according to a preset storage rule. For example, the target storage rule may include, but is not limited to, a mapping relationship between a report and a database table, a mapping relationship between a data item in a report and a data item in a database table, and the like.
After the target storage plug-in successfully stores the target data to the target storage system, the storage path of the target data can be returned. Therefore, when the report needs to be checked, updated, modified or deleted in the future, the report can be searched according to the storage path.
Thus, through the data access component 310, the data parsing component 320, the data verification component 330, and the data storage component 340, the data submission system 300 successfully collects data submitted by the user 110 into the data storage system.
By employing a hierarchical, componentized, and configured architectural design, the data messaging system 300 retains a core execution framework of the following components: the data access component 310 receives the message from the sender; a data parsing and converting component 320, which converts the message into parsing data in a target format; a data verification component 330 for verifying the analysis data; and a data storage component 340 for storing the analysis data in the target format. The frame program eliminates the difference of the data formats of the input ends, and the frame program uniformly converts the data format of the input ends into a preset target format and processes the data in the preset target format no matter what the format of the data of the input ends is. The data reporting system 300 described in the present application can access different plug-ins for messages of different formats through a plug-in architecture design, process the messages of different formats into data of a uniform format through the plug-in, and then access the data into the core execution framework program for data processing. By adopting the mode, when the data reported by the reporting end is in a new data format, workers only need to write a specific plug-in unit for the message in the new data format without modifying the core execution framework program. Therefore, different report template formats, different data formats, different reporting modes, different verification modes and different storage modes can be flexibly and extendably supported, so that the method can be widely reused in various scenes in various fields, and the defects of long development period, labor and financial consumption, difficulty in maintenance and the like caused by customized development are overcome.
In some embodiments, the data submission system 300 can further include a configuration component 350, a submission management component 360, an authority management component 370, and a data query component 380 to perform effective management and authority isolation on the collected huge data, further ensuring efficient use and security of the data.
In some embodiments, the data submission system 300 can also include a submission management component 360. The submission management component 360 can manage the submission status of the report. For example, the status of a submission to a report may be divided into "submitted" and "not submitted". The submission management component 360 can manage the status of the report. The report management component 360 can also issue a message notification to the reporting main body according to the report status of the report. The submission management component 360 may also manage template issuance, template version, submission data, late submissions, missed submissions, and press submissions for a multi-terminal submission facility.
In some embodiments, the messaging system 300 may also include a configuration component 350. The configuration component 350 includes at least one configuration interface. The at least one configuration interface is configured to receive configuration information. The user 110 may configure the parameters of the posted data via a configuration interface displayed on the terminal 130. For example, user 110 may configure the report type, data format, reporting mechanism, etc. of the reported data via a configuration interface. The report submitting mode, the analysis rule, the verification rule, the storage type, the authority, the Excel template, the API, the filled data item, the global parameter and the like of the report can be configured by the personnel or the developer of the monitoring organization through the configuration interface. The configuration interface provides a visual interface to the user, making the system 300 user friendly.
In some embodiments, the datagram system 300 may include a rights management component 370. Rights management component 370 may include a functional rights management module and a data rights management module.
The functional rights management component enables different users to access only the functions that are granted rights by assigning different roles to the users. For example, a bank in city a, as a regulatory agency, has the right of jurisdiction only for each district/county branch in city a within its jurisdiction. Therefore, the function authority of a certain bank in the city A can be configured through the function authority management module, so that the certain bank in the city A can only issue a report sending task and/or a report template to the branch office with the function authority.
The data rights management module may manage the rights of the data received by the data delivery system 300. The data authority management module can ensure data isolation between different organizations or departments. For example, by setting the data authority, the organization a can only view and change the data of the report issued by the organization a, and the organization B can only view and change the data of the report issued by the organization B.
In some embodiments, the data submission system 300 can also include a data query component 380. Data query component 380 supports querying of raw submission data, parsed stored data, and statistics of the data. For example, a report is uploaded by the regulatory body through the reporting system 300, and the reporting system 300 stores the report in a file storage system. The supervised person and/or the supervising authority person may query the reported data through the foreground of the data reporting system 300 displayed on the terminal 130. For example, the administrative staff may perform detail query, report query or statistical query on the delivery data through the data query module in the data delivery system 300.
In summary, the present application provides a data reporting system. The data reporting system realizes the abstraction and multiplexing of functions by carrying out aggregation and classification on different functions according to layers. The data system expands the functions of the product in a plug-in mode without modifying the original product code. The data reporting system packs the functional modules into components for multiplexing. The data reporting system can support the check and reporting of reports of different formats in a visual configuration modification mode, and code development is avoided.
The data submission system can flexibly and extendably support different report template formats, different data formats, different submission modes, different verification modes and different storage modes by adopting layered, plug-in, componentized and configured architecture design, so that the data submission system can be widely reused in various scenes in various fields, and the defects of long development period, labor and financial consumption, difficulty in maintenance and the like caused by customized development are avoided.
The application also provides a data reporting method, which comprises the following steps: receiving a message from a transmitting end through at least one data access plug-in, wherein the message comprises target data; loading a target analysis plug-in according to a preset analysis rule, analyzing the target data and converting the target data into a target format; loading a target verification plug-in according to a preset verification rule, and verifying the analyzed target data; and loading a target storage plug-in according to a preset storage rule, and storing the target data passing the verification to a target storage system.
For the specific steps of the data reporting method, reference may be made to the description of the data reporting system 300, and for brevity, the detailed description is omitted here.
In conclusion, upon reading the present detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure can be presented by way of example only, and not limitation. Those skilled in the art will appreciate that the present application is intended to cover various reasonable variations, adaptations, and modifications of the embodiments described herein, although not explicitly described herein. Such alterations, improvements, and modifications are intended to be suggested by this application and are within the spirit and scope of the exemplary embodiments of the application.
Furthermore, certain terminology has been used in this application to describe embodiments of the application. For example, "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various portions of this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined as suitable in one or more embodiments of the application.
It should be appreciated that in the foregoing description of embodiments of the present application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of such feature. Alternatively, various features may be dispersed throughout several embodiments of the application. This is not to be taken as an admission that any of the features of the claims are essential, and it is fully possible for a person skilled in the art to extract some of them as separate embodiments when reading the present application. That is, embodiments in the present application may also be understood as an integration of multiple sub-embodiments. And each sub-embodiment described herein is equally applicable to less than all features of a single foregoing disclosed embodiment.
In some embodiments, numbers expressing quantities or properties useful for describing and claiming certain embodiments of the present application are to be understood as being modified in certain instances by the terms "about", "approximately" or "substantially". For example, "about", "approximately" or "substantially" may mean a ± 20% variation of the value it describes, unless otherwise specified. Accordingly, in some embodiments, the numerical parameters set forth in the written description and attached claims are approximations that may vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, numerical parameters should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the embodiments of the application are approximations, the numerical values set forth in the specific examples are reported as precisely as possible.
Each patent, patent application, publication of a patent application, and other material, such as articles, books, descriptions, publications, documents, articles, and the like, cited herein is hereby incorporated by reference. All matters hithertofore set forth herein except as related to any prosecution history, may be inconsistent or conflicting with this document or any prosecution history which may have a limiting effect on the broadest scope of the claims. Now or later associated with this document. For example, if there is any inconsistency or conflict in the description, definition, and/or use of terms associated with any of the included materials with respect to the terms, descriptions, definitions, and/or uses associated with this document, the terms in this document are used.
Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of the present application. Other modified embodiments are also within the scope of the present application. Accordingly, the disclosed embodiments are presented by way of example only, and not limitation. Those skilled in the art may implement the present application in alternative configurations according to the embodiments of the present application. Thus, embodiments of the present application are not limited to those embodiments described with precision in the application.
Claims (13)
1. A plug-in delivery system comprising:
the plug-in library comprises a plurality of plug-ins;
the data access component is configured to receive a message from a transmitting end, and the message comprises target data;
a data parsing conversion component comprising at least one first plug-in interface and configured to:
receiving the target data from the data access component,
selecting a target parsing plug-in from the at least one parsing plug-in,
sending the target data to the target analysis plug-in through the at least one first plug-in interface to analyze the target data,
receiving parsed data of the target data from the target parsing plug-in through the at least one first plug-in interface, and
converting the format of the analysis data into a target format;
a data verification component configured to verify the parsed data; and
a data storage component configured to store the parsed data to a target storage system of a storage medium.
2. The messaging system of claim 1, wherein each of the at least one parsing plug-in comprises:
the data input format of the data input interface corresponds to the data format of the output end of the first plug-in interface; and
and the output data format of the data output interface corresponds to the data format of the input data end of the first plug-in interface.
3. The messaging system of claim 2, wherein said at least one parsing plug-in includes at least one of an Excel parsing engine, a CSV streaming parsing engine, an API data transformation engine, and an XML/DAT format parsing engine.
4. The messaging system of claim 1, wherein the data verification component comprises at least one second plug-in interface and is configured to:
selecting a target verification plug-in from at least one data verification plug-in;
sending the analysis data to the target inspection plug-in through the at least one second plug-in interface; and
receiving a verification result from the target verification plug-in through the at least one second plug-in interface.
5. The messaging system of claim 4, wherein the target verification plug-in verifies the parsed data according to predetermined verification rules, wherein the verification rules include at least one of format verification rules, inline rule verification rules, uniqueness verification rules, script verification rules, and inter-table verification rules.
6. The messaging system of claim 4, wherein the at least one second plug-in interface selects the target verification plug-in from the at least one verification plug-in based on configuration information.
7. The messaging system of claim 1, wherein the data storage component comprises at least one third plug-in interface and is configured to:
selecting a target storage plug-in from at least one data storage plug-in;
sending the verified data to the target storage plug-in through the at least one third plug-in interface; and
receiving a storage result from the target storage plug-in through the at least one third plug-in interface.
8. The data reporting system of claim 7, wherein the at least one storage plug-in comprises at least one of a MySQL database storage plug-in, an OceanBase database storage plug-in, a HIVE database storage plug-in, an Oracel database storage plug-in, a SQLServer database storage plug-in, a local file system storage plug-in, a network file system NFS storage plug-in, a Hadoop file system HDFS storage plug-in, and an object storage OSS storage plug-in; and
the storage result includes a storage path of the target data.
9. The messaging system of claim 7, wherein the at least one third plug-in interface selects the target storage plug-in from the at least one storage plug-in based on configuration information.
10. The messaging system of claim 1, wherein the data access component comprises at least one fourth plug-in interface and is configured to:
selecting a target access plug-in from at least one data access plug-in;
sending the message to the target access plug-in through the at least one fourth plug-in interface; and
and receiving the data of the target data after decryption and integrity verification from the target access plug-in through the at least one fourth plug-in interface.
11. The datagram system as claimed in claim 10, wherein the at least one data access plug-in includes at least one of a manual web upload plug-in, a pre-processor post-verification automatic upload plug-in, an OpenAPI call upload plug-in, and a web fill plug-in.
12. The data delivery system of claim 1, further comprising:
a configuration component comprising at least one configuration interface configured to receive configuration information.
13. A method of data reporting, comprising:
receiving a message from a transmitting end through at least one data access plug-in, wherein the message comprises target data;
loading a target analysis plug-in according to a preset analysis rule, analyzing the target data and converting the target data into a target format;
loading a target verification plug-in according to a preset verification rule, and verifying the analyzed target data; and
and loading the target storage plug-in according to a preset storage rule, and storing the verified target data to a target storage system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010799238.0A CN111782718B (en) | 2020-08-11 | 2020-08-11 | Plug-in data reporting system and data reporting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010799238.0A CN111782718B (en) | 2020-08-11 | 2020-08-11 | Plug-in data reporting system and data reporting method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111782718A true CN111782718A (en) | 2020-10-16 |
| CN111782718B CN111782718B (en) | 2023-12-29 |
Family
ID=72761822
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010799238.0A Active CN111782718B (en) | 2020-08-11 | 2020-08-11 | Plug-in data reporting system and data reporting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111782718B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112286918A (en) * | 2020-10-23 | 2021-01-29 | 佳都新太科技股份有限公司 | Method and device for fast access conversion of data, electronic equipment and storage medium |
| CN112463885A (en) * | 2020-11-26 | 2021-03-09 | 北京宏景世纪软件股份有限公司 | Data synchronization implementation method and centralized data system |
| CN112948429A (en) * | 2021-02-02 | 2021-06-11 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN116049293A (en) * | 2023-03-23 | 2023-05-02 | 北京沐融信息科技股份有限公司 | Method, device, equipment and medium for realizing analysis of CSV file based on database configuration |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020078216A1 (en) * | 2000-12-18 | 2002-06-20 | Igor Pankovcin | Method and system for processing data records having multiple formats |
| WO2007068175A1 (en) * | 2005-12-13 | 2007-06-21 | Huawei Technologies Co., Ltd. | A system and method for triggering the rule system |
| CN103150647A (en) * | 2013-04-10 | 2013-06-12 | 中国农业银行股份有限公司 | Report file processing method, server and system |
| CN105469188A (en) * | 2015-11-16 | 2016-04-06 | 国网山东省电力公司经济技术研究院 | Distribution network planning data online intelligent analysis system and distribution network planning data online intelligent analysis method |
| US20190155801A1 (en) * | 2017-08-16 | 2019-05-23 | Walmart Apollo, Llc | Systems and methods for distributed data validation |
| CN109918403A (en) * | 2019-02-02 | 2019-06-21 | 中国银行股份有限公司 | Data verification method, device, computer equipment and storage medium |
| CN110163735A (en) * | 2019-04-09 | 2019-08-23 | 平安科技（深圳）有限公司 | Concerning taxes data processing method, device, computer equipment and storage medium |
| WO2019196304A1 (en) * | 2018-04-09 | 2019-10-17 | 平安科技（深圳）有限公司 | Electronic apparatus, credit feedback message parsing method, and storage medium |
-
2020
- 2020-08-11 CN CN202010799238.0A patent/CN111782718B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020078216A1 (en) * | 2000-12-18 | 2002-06-20 | Igor Pankovcin | Method and system for processing data records having multiple formats |
| WO2007068175A1 (en) * | 2005-12-13 | 2007-06-21 | Huawei Technologies Co., Ltd. | A system and method for triggering the rule system |
| CN103150647A (en) * | 2013-04-10 | 2013-06-12 | 中国农业银行股份有限公司 | Report file processing method, server and system |
| CN105469188A (en) * | 2015-11-16 | 2016-04-06 | 国网山东省电力公司经济技术研究院 | Distribution network planning data online intelligent analysis system and distribution network planning data online intelligent analysis method |
| US20190155801A1 (en) * | 2017-08-16 | 2019-05-23 | Walmart Apollo, Llc | Systems and methods for distributed data validation |
| WO2019196304A1 (en) * | 2018-04-09 | 2019-10-17 | 平安科技（深圳）有限公司 | Electronic apparatus, credit feedback message parsing method, and storage medium |
| CN109918403A (en) * | 2019-02-02 | 2019-06-21 | 中国银行股份有限公司 | Data verification method, device, computer equipment and storage medium |
| CN110163735A (en) * | 2019-04-09 | 2019-08-23 | 平安科技（深圳）有限公司 | Concerning taxes data processing method, device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 元晶;: "一种基于三层架构的财务信息报送平台", 甘肃农业, no. 11 * |
| 曾庆妍;王浩;: "监管报送数据质量平台的设计与实现", 信息技术与信息化, no. 01 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112286918A (en) * | 2020-10-23 | 2021-01-29 | 佳都新太科技股份有限公司 | Method and device for fast access conversion of data, electronic equipment and storage medium |
| CN112286918B (en) * | 2020-10-23 | 2023-07-14 | 佳都科技集团股份有限公司 | Method and device for fast access conversion of data, electronic equipment and storage medium |
| CN112463885A (en) * | 2020-11-26 | 2021-03-09 | 北京宏景世纪软件股份有限公司 | Data synchronization implementation method and centralized data system |
| CN112948429A (en) * | 2021-02-02 | 2021-06-11 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN112948429B (en) * | 2021-02-02 | 2024-04-26 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN116049293A (en) * | 2023-03-23 | 2023-05-02 | 北京沐融信息科技股份有限公司 | Method, device, equipment and medium for realizing analysis of CSV file based on database configuration |
| CN116049293B (en) * | 2023-03-23 | 2024-02-13 | 北京沐融信息科技股份有限公司 | Method, device, equipment and medium for realizing analysis of CSV file based on database configuration |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111782718B (en) | 2023-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111782718A (en) | Plug-in data reporting system and data reporting method | |
| US11397744B2 (en) | Systems and methods for data storage and processing | |
| CN105493121B (en) | System and method for controlling electronic communication | |
| US8806656B2 (en) | Method and system for secure and selective access for editing and aggregation of electronic documents in a distributed environment | |
| Kostick-Quenet et al. | How NFTs could transform health information exchange | |
| CN107833009A (en) | Personalized checking method, device, storage medium and the terminal of Claims Resolution order | |
| CN102722533A (en) | Method and system for displaying and filling report | |
| JP7161538B2 (en) | Systems, apparatus and methods for processing and managing web traffic data | |
| CN108681674B (en) | Report module creating method and device, computer device and storage medium | |
| CN112434015B (en) | Data storage method and device, electronic equipment and medium | |
| US20200409543A1 (en) | Application custom property framework | |
| CN110414259A (en) | A kind of method and apparatus for constructing data element, realizing data sharing | |
| US11544669B2 (en) | Computing framework for compliance report generation | |
| CN115544183A (en) | Data visualization method and device, computer equipment and storage medium | |
| US20220358185A1 (en) | Traversing data structures for compliance | |
| Portela et al. | Using digital technologies in response to the COVID-19 pandemic in Portugal | |
| Kawu et al. | FAIR Guidelines and data regulatory framework for digital health in Nigeria | |
| CN107392560A (en) | A kind of Excel list datas issue acquisition method and system based on internet | |
| CN113297179A (en) | Method and device for processing supervision submission data | |
| Tadele Taye et al. | FAIR Equivalency with regulatory framework for digital health in Ethiopia | |
| CN110362493A (en) | Electronic equipment response method and device, electronic equipment and readable storage medium storing program for executing | |
| US20220236957A1 (en) | Dynamic application builder for multidimensional database environments | |
| CN115022339A (en) | Block chain based supply chain tracing method and system and electronic equipment | |
| CN114385722A (en) | Interface attribute consistency checking method and device, electronic equipment and storage medium | |
| CN114611861A (en) | Enterprise resource informatization management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |