CN117951135A - Data processing method, device, computer equipment and storage medium - Google Patents
Data processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117951135A CN117951135A CN202410011349.9A CN202410011349A CN117951135A CN 117951135 A CN117951135 A CN 117951135A CN 202410011349 A CN202410011349 A CN 202410011349A CN 117951135 A CN117951135 A CN 117951135A
- Authority
- CN
- China
- Prior art keywords
- database table
- scheme
- database
- data
- measuring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 15
- 238000013507 mapping Methods 0.000 claims abstract description 88
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000004590 computer program Methods 0.000 claims abstract description 25
- 238000005259 measurement Methods 0.000 claims description 181
- 238000012545 processing Methods 0.000 claims description 29
- 238000012795 verification Methods 0.000 claims description 17
- 230000004044 response Effects 0.000 claims description 8
- 238000012360 testing method Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 description 11
- 230000008878 coupling Effects 0.000 description 9
- 238000010168 coupling process Methods 0.000 description 9
- 238000005859 coupling reaction Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 239000002344 surface layer Substances 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 238000013500 data storage Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 239000010410 layer Substances 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000035515 penetration Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to a data processing method, apparatus, computer device, storage medium and computer program product. The method comprises the following steps: responding to input operation, and acquiring a first database table corresponding to the current measuring and calculating scheme; responding to the measuring instruction, and acquiring a second database table corresponding to the current measuring scheme; determining a third database table of the current measuring and calculating scheme based on the first database table and the intermediate mapping table; and determining a measuring result based on the second database table and the third database table of the current measuring scheme. By adopting the method, the field information can be flexibly added in the database table, and the flexibility of the system is improved.
Description
Technical Field
The present application relates to the field of computer technology, and in particular, to a data processing method, apparatus, computer device, storage medium, and computer program product.
Background
With the development of computer technology, database technology has emerged, which is a core technology in the field of information systems. Databases are data sets stored together in a manner that can be shared by multiple users, with as little redundancy as possible, independent of the application.
In the conventional technology, a system includes a plurality of modules, each module needs massive data support, data in each module is stored in a database table, and data association between modules, namely, association between database tables storing data.
However, in the current data association between modules in a system, the data between modules often show strong coupling characteristics, and flexible adjustment and modification of the data in the modules cannot be achieved.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that enable flexibility.
In a first aspect, the present application provides a data processing method, including:
responding to input operation, and acquiring a first database table corresponding to the current measuring and calculating scheme;
responding to the measuring instruction, and acquiring a second database table corresponding to the current measuring scheme;
determining a third database table of the current measurement scheme based on the first database table and the intermediate mapping table;
and determining a measuring result based on the second database table and the third database table of the current measuring scheme.
In one embodiment, the generating manner of the intermediate mapping table includes:
Acquiring a history scheme information table, a history first database table and a history third database table corresponding to a history test scheme;
generating a first intermediate mapping table based on the historical first database table and the historical scheme information table, wherein fields in the first intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table;
Generating a second intermediate mapping table based on the historical third database table and the historical scheme information table, wherein fields in the second intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table;
The determining a third database table of the current measurement scheme based on the first database table and the middle mapping table corresponding to the current measurement scheme includes:
determining a scheme information table corresponding to the current measuring scheme based on a first database table corresponding to the current measuring scheme and the first intermediate mapping table;
and determining a third database table corresponding to the current measuring and calculating scheme based on the scheme information table corresponding to the current measuring and calculating scheme and the second intermediate mapping table.
In one embodiment, the third database table includes first measurement data; the second database table includes second measurement data; the determining a measurement result based on the second database table and the third database table of the current measurement scheme includes:
determining a key value pair corresponding to the first measurement data based on the third database table;
Determining a key value pair corresponding to the second measurement data based on the second database table;
Determining a measurement result according to the key value pair corresponding to the first measurement data and the key value pair corresponding to the second measurement data.
In one embodiment, before the responding to the measurement instruction and obtaining the second database table corresponding to the current measurement scheme, the method further includes:
Acquiring initial second measurement data and storing the initial second measurement data in cloud storage;
Acquiring initial second measurement data from the cloud storage through ETL service;
Invoking the ETL service, and performing verification processing on the initial second measurement data to obtain second measurement data, and storing the second measurement data in the second database table.
In one embodiment, the invoking the ETL service performs verification processing on the initial second measurement data to obtain second measurement data, and stores the second measurement data in the second database table, including:
invoking ETL service, checking the field and field value of the initial second measurement data, determining the field and field value as the second measurement data after successful check,
Traversing the fields of each second database table according to the fields of the second measuring and calculating data;
After the fields of the second measuring and calculating data are successfully matched with the fields of each second database table, the second measuring and calculating data are stored in the corresponding second database tables.
In one embodiment, the invoking the ETL service performs verification processing on the initial second measurement data to obtain second measurement data, and after storing the second measurement data in the second database table, the method further includes:
and returning the storage state of the second measuring and calculating data through the message queue.
In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:
The first acquisition module is used for responding to the input operation and acquiring a first database table corresponding to the current measuring and calculating scheme;
The second acquisition module is used for responding to the measuring and calculating instruction and acquiring a second database table corresponding to the current measuring and calculating scheme;
The table determining module is used for determining a third database table of the current measuring and calculating scheme based on the first database table and the intermediate mapping table;
the result determining module is used for determining a measuring and calculating result based on the second database table and the third database table of the current measuring and calculating scheme.
In a third aspect, the present application provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when the processor executes the computer program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method described above.
In a fifth aspect, the application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method described above.
The data processing method, the data processing device, the computer equipment, the storage medium and the computer program product are characterized in that a first database table corresponding to a current measuring and calculating scheme is obtained by responding to input operation firstly; secondly, responding to the measuring and calculating instruction, and acquiring a second database table corresponding to the current measuring and calculating scheme; then, determining a third database table of the current measuring and calculating scheme based on the first database table and the intermediate mapping table; and finally, determining a measuring result based on the second database table and the third database table of the current measuring scheme. The data are transmitted in the database tables through the preset intermediate mapping tables, various data are processed and stored in the intermediate mapping tables in a unified format, namely, the first database tables transmit the data through the intermediate mapping tables, so that the logic coupling relation between the first database tables is reduced, the flexibility of calling the first database tables is improved, the third database tables are determined, the measuring and calculating result is determined through the second database tables and the third database tables, the strong coupling between the database tables is reduced, the coupling relation between the data fields and other database tables is not needed to be considered when the data fields are increased, the flexible field information increase in the database tables is realized, and the flexibility of a system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for those skilled in the art.
FIG. 1 is a diagram of an application environment for a data processing method in one embodiment;
FIG. 2 is a flow diagram of a data processing method in one embodiment;
FIG. 3 is a flowchart illustrating a method for generating an intermediate mapping table according to another embodiment;
FIG. 4 is a flow chart of determining a third database table of the current measurement scheme based on the first database table and the intermediate mapping table corresponding to the current measurement scheme in one embodiment;
FIG. 5 is a schematic diagram illustrating a use procedure of the intermediate mapping table in one embodiment;
FIG. 6 is a flow chart of determining a measurement result based on a second database table and a third database table of the current measurement scheme in one embodiment;
FIG. 7 is a flow chart of importing second measurement data according to one embodiment;
FIG. 8 is a schematic diagram of an architecture for importing second measurement data in one embodiment;
FIG. 9 is a schematic flow diagram of invoking an ETL service, verifying initial second measured data to obtain second measured data, and storing the second measured data in a second database table, in one embodiment;
FIG. 10 is a flow chart of import checking to save second measured data in one embodiment;
FIG. 11 is a flow chart of import checking to save second measured data in one embodiment;
FIG. 12 is a block diagram of a data processing apparatus in one embodiment;
fig. 13 is an internal structural view of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The data processing method provided by the embodiment of the application can be applied to an application environment shown in figure 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The server responds to input operation to acquire a first database table corresponding to the current measuring and calculating scheme; responding to the measuring instruction, and acquiring a second database table corresponding to the current measuring scheme; determining a third database table of the current measuring and calculating scheme based on the first database table and the intermediate mapping table; and determining a measuring result based on the second database table and the third database table of the current measuring scheme. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.
In an exemplary embodiment, as shown in fig. 2, a data processing method is provided, and an example of application of the method to the server in fig. 1 is described, including the following steps S202 to S208. Wherein:
Step S202, responding to input operation, and acquiring a first database table corresponding to the current measuring and calculating scheme.
The input operation may be an input operation for the user of the measuring scheme, or an input operation mainly comprising other devices. The input operation may be a selection operation by the user in a drop-down box on the page, or may be an input operation by the user in a text box. The first database table may be a database configuration table, which is selected from the database based on a user input operation, thereby forming a database table.
Optionally, the user may issue a user input operation instruction, for example, an operation selected by a drop-down box in the page, or an operation of inputting data in a text box, and in response, the server may generate a first database table corresponding to the current measurement scheme in response to the user input operation. For example, a user selects an operation in a drop-down box in a page and inputs data in a text box, and the server responds to a user input operation instruction and generates a first database table formed by the user through configuration based on information selected by the user drop-down box and information input in the text box. The server saves a first database table generated according to a user input operation as template information.
Further, the user may also select template information in the page, where the template information includes the database configuration table that has been generated. When a user selects template information in a page, the server responds to the operation of selecting the template information by the user to acquire a first database table corresponding to the template information.
Optionally, the server may further accept the user to edit and delete the data in the template information created by the user, and generate a database configuration table according to the user's editing operation; the server may also generate a database configuration table based on the user's delete operation.
Step S204, responding to the measuring instruction, and acquiring a second database table corresponding to the current measuring scheme.
The second database table is used for acquiring a database fact table stored in the database, namely the second database table, according to the user measuring and calculating instruction and information carried in the measuring and calculating instruction.
Optionally, the server responds to the user measuring and calculating instruction, and also responds to measuring and calculating instructions issued by other devices, and a corresponding second database table is obtained from the database according to the identification of the current measuring and calculating scheme, wherein the second database table comprises information such as field description, field name, field type and the like. The data stored in the second database table is uploaded to the database before the second database table corresponding to the current measuring scheme is acquired in response to the measuring instruction, or may be a data warehouse composed of a plurality of databases.
Optionally, the user clicks a submit button in the page, sends the measurement request to the server, and the server receives the sending request and obtains the corresponding database fact table and the data recorded in the database fact table from the database based on the test request.
Step S206, determining a third database table of the current measuring scheme based on the first database table and the intermediate mapping table.
Wherein the intermediate mapping table is pre-generated according to the history data and the history database table. Wherein the generation of the intermediate mapping table is based on a plurality of history database tables.
Optionally, the server performs sorting processing on the obtained various data and stores the data in the intermediate mapping table in a uniform format, for example, the intermediate mapping table may be consistent with a storage mode of the first database table or may be consistent with a storage mode of the second database table. For example, the intermediate mapping table includes a field chinese name, a field name, and a field type, where the field type may be integer type, floating point type, fixed point type, and the field type may further include char, varchar, integer types.
Optionally, the server obtains a first database table and an intermediate mapping table, for example, a first intermediate mapping table, where the first intermediate mapping table records a unique field in the first database table and a unique field in the third database table, determines a unique field of the third database table of the current measurement scheme according to the unique field in the first database table and the unique field in the third database table recorded in the first intermediate mapping table, and determines the third database table according to the unique field of the third database table. The intermediate mapping table may be one table or a plurality of tables.
Step S208, determining a measuring result based on the second database table and the third database table of the current measuring scheme.
Optionally, the server calculates based on a second database table, that is, a database configuration table, of the current measurement scheme, and a third database table, that is, data stored in a database fact table, where the database configuration table records analog data of the current measurement scheme, the database fact table records real data of the current measurement scheme, and the real data may be imported and stored in the database fact table in advance, and the measurement result is determined according to the analog data of the current measurement scheme recorded in the database configuration table and the real data of the current measurement scheme recorded in the database fact table.
In the data processing method, a first database table corresponding to a current measuring and calculating scheme is acquired by responding to user input operation; secondly, responding to the measuring and calculating instruction, and acquiring a second database table corresponding to the current measuring and calculating scheme; then, determining a third database table of the current measuring and calculating scheme based on the first database table and the intermediate mapping table; and finally, determining a measuring result based on the second database table and the third database table of the current measuring scheme. The data are transmitted in the database tables through the preset intermediate mapping tables, various data are processed and stored in the intermediate mapping tables in a unified format, namely, the first database tables transmit the data through the intermediate mapping tables, so that the logic coupling relation between the first database tables is reduced, the flexibility of calling the first database tables is improved, the third database tables are determined, the measuring and calculating result is determined through the second database tables and the third database tables, the strong coupling between the database tables is reduced, the coupling relation between the data fields and other database tables is not needed to be considered when the data fields are increased, the flexible field information increase in the database tables is realized, and the flexibility of a system is improved.
In an exemplary embodiment, as shown in fig. 3, the generation manner of the intermediate mapping table includes steps S302 to S306. Wherein:
Step S302, a history scheme information table, a history first database table and a history third database table corresponding to the history measuring and calculating scheme are obtained.
The history scheme information table is generated according to the history measuring scheme information. The first historical database table is a database configuration table generated according to the historical operation input, for example, the first historical database table can be a bottom-layer asset table of the history or a surface-layer asset table of the history. The historical third database table may be a third database table derived from the historical first database table and the intermediate mapping table.
Optionally, the server acquires a history scheme information table, a history first database table and a history third database table corresponding to the history measurement scheme 1. The history scheme information table is a history scheme information table generated according to history measurement scheme information, and can include field information such as a scheme universal unique identification code UUID (Universally Unique Identifier, UUID), a scheme version and the like, and the history scheme information table is shown in table 1. The historical first database table comprises a scheme UUID, a surface layer asset UUID and a penetration condition, wherein the penetration condition can be determined by a user according to a drop-down box on a page, and is shown in table 2. The history third database table includes field information of record uuid, calculation time point, job number, etc., as shown in table 3. The server may also obtain other history measurement schemes, such as a history scheme information table, a history first database table, and a history third database table corresponding to the history measurement scheme 2.
TABLE 1
TABLE 2
TABLE 3 Table 3
It should be noted that, the field information in the history scheme information table, the history first database table, and the history third database table corresponding to the history test scheme is not limited.
Step S304, a first intermediate mapping table is generated based on the history first database table and the history scheme information table, wherein fields in the first intermediate mapping table are associated with fields in the history first database table and fields in the history scheme information table.
Optionally, the server generates the first intermediate mapping table based on fields in a plurality of historical first database tables of different historical measurement schemes and fields in a corresponding historical scheme information table of the historical measurement scheme.
Further, the server is based on a plurality of historical first database tables of the historical measurement scheme 1, wherein the historical first database tables may include a surface layer asset table, a bottom layer asset table such as table 2, and other tables. According to the scheme UUID, the surface layer asset UUID and the history scheme information table in the bottom layer asset table 2, such as id and scheme UUID in table 1, the scheme UUID in table 2 is associated with the scheme UUID in table 1, so as to generate a first intermediate mapping table, such as shown in table 4.
TABLE 4 Table 4
In step S306, a second intermediate mapping table is generated based on the historical third database table and the historical schema information table, and fields in the second intermediate mapping table are associated with fields in the historical first database table and fields in the historical schema information table.
Optionally, the server generates the second intermediate mapping table based on fields in a historical third database table of the different historical measurement schemes and fields in a corresponding historical scheme information table of the historical measurement schemes.
Further, the server is based on a history third database table of history measurement scheme 1, where the history third database table may include a record table of history measurement scheme 1 such as table 3, and may include other tables. According to the record table of history measurement scheme 1 such as id in table 3, record uuid, and the history scheme information table such as id in table 1, scheme uuid, the id in table 3 is associated with the id in table 1, thereby generating a second intermediate map table as shown in table 5.
TABLE 5
In this embodiment, the first intermediate mapping table and the second intermediate mapping table are obtained through the history scheme information table, the history first database table and the history third database table of the history measurement scheme. And storing the field information in the first intermediate mapping table and the second intermediate mapping table so that the subsequent first database table can transmit data through the first intermediate mapping table and the second intermediate mapping table to obtain a third database table, thereby reducing the coupling between tables.
In the above embodiment, as shown in fig. 4, the third database table of the current measurement scheme is determined based on the first database table and the intermediate mapping table corresponding to the current measurement scheme, which includes steps S402 to S406. Wherein:
step S402, determining a scheme information table corresponding to the current measuring scheme based on a first database table and a first intermediate mapping table corresponding to the current measuring scheme.
Optionally, the server acquires a first database table corresponding to the current measurement and calculation scheme, and determines a scheme information table corresponding to the current measurement and calculation scheme according to the first intermediate mapping table obtained by storing the historical data.
Further, the server acquires a first database table corresponding to the current measuring and calculating scheme, for example, the first database table is a surface layer asset table, and according to the association relationship between the ID, the scheme uuid and the surface layer asset uuid field recorded in the first database table and the ID, the scheme uuid and the surface layer asset uuid field in the first intermediate mapping table, the scheme information table corresponding to the current measuring and calculating scheme is determined.
Step S404, determining a third database table corresponding to the current measuring and calculating scheme based on the scheme information table corresponding to the current measuring and calculating scheme and the second intermediate mapping table.
Optionally, the server acquires a scheme information table corresponding to the current measurement scheme, and determines a third database table corresponding to the current measurement scheme according to the second intermediate mapping table obtained by storing the historical data.
Further, the server obtains a solution information table corresponding to the current measurement solution, for example, a solution information table corresponding to the measurement solution 1, where the table may include fields such as ID, solution uuid, solution version, whether a template is used for distinguishing whether the solution is template information or not, if so, a solution of the user can be quickly built, and whether the current solution is the measurement solution 1 is determined by the solution uuid. And determining the record ID in the second intermediate mapping table according to the ID recorded in the scheme information table, whether the scheme uuid field is matched with the ID in the second intermediate mapping table and whether the scheme uuid associated field is matched with the ID in the second intermediate mapping table, and determining a third database table corresponding to the current measuring scheme, namely the current record table according to the record ID.
Optionally, as shown in fig. 5, the server determines the scheme information table of the current measurement scheme by acquiring the scheme UUID in the first database table, the first database table 1UUID field, the scheme UUID in the first intermediate mapping table, and the first database table 1UUID field. And the server determines a UUID field according to the UUID of the scheme in the scheme information table of the current measuring scheme, the UUID of the scheme version field and the UUID of the scheme in the second intermediate mapping table, so as to obtain a third database table, namely the record table of the current measuring scheme, according to the UUID field of the record.
In this embodiment, the fields in the first database table perform data transfer through the fields pre-stored in the first intermediate mapping table and the second intermediate mapping table, so that the third database table can be obtained according to the intermediate mapping table by inputting the field information of the first database table, and the coupling relationship between the first database table and the third database table is reduced, thereby improving the flexibility when the field information is increased.
In one exemplary embodiment, as shown in FIG. 6, the third database table includes first measured data; the second database table includes second measurement data; determining a measurement result based on the second database table and the third database table of the current measurement scheme, including steps S602 to S606. Wherein:
Step S602, determining a key value pair corresponding to the first measurement data based on the third database table.
The hash is also called a key value pair, and is a mapping set of a string type key and a value, for example, the key: value (key: value). The fundamental unit of HashMap stored data is a key-value pair, where each key is unique and each key is associated with a value.
Optionally, the server obtains the first measurement data in the third database table, and converts the first measurement data into a key value pair form. The server directly finds the value stored in the memory through the key (key), so that the speed of accessing the data is improved.
Step S604, determining a key value pair corresponding to the second measurement data based on the second database table.
Optionally, the server obtains second measurement data in the third database table, and converts the second measurement data into a key value pair form.
Step S606, determining a measuring result according to the key value pair corresponding to the first measuring data and the key value pair corresponding to the second measuring data.
Optionally, the server determines the value of each first measurement according to the key value pair corresponding to the first measurement data; determining the value of each second measurement and calculation according to the key value pair corresponding to the second measurement and calculation data, substituting a preset calculation formula according to the value of each first measurement and calculation and the value of each second measurement and calculation, and determining the measurement and calculation result.
In this embodiment, the first measurement data and the second measurement data are stored in a key value pair manner, and the data are transmitted in a key value pair manner, so that flexibility and efficiency of data calculation can be improved, and an effect of supporting dynamic expansion of fields can be achieved.
In an exemplary embodiment, as shown in fig. 7, before the second database table corresponding to the current measurement scheme is obtained in response to the measurement instruction, steps S702 to S706 are further included. Wherein:
Step S702, obtaining initial second measurement data and storing the initial second measurement data in cloud storage.
Cloud storage is a mode of online storage (Cloud storage), in which data is stored on multiple virtual servers, typically hosted by third parties, rather than on dedicated servers.
Optionally, the server obtains initial second measurement data, which may be stored in XLSX format or XLSB format. The server acquires the initial second measurement data stored in XLSX, and uploads the initial second measurement data to the cloud storage.
Step S704, acquiring initial second measurement data from cloud storage through the ETL service.
The ETL is a process of loading (Load) data of a business system into a data warehouse after extraction (extraction) and cleaning and conversion (transformation), and aims to integrate scattered, scattered and non-uniform data.
Optionally, the server obtains initial second measurement data from the cloud storage through the ETL service.
Step S706, the ETL service is invoked, verification processing is carried out on the initial second measurement data, the second measurement data is obtained, and the second measurement data is stored in a second database table.
Optionally, the server invokes the ETL service through the JAVA programming language, performs verification processing on the initial second measurement data to obtain second measurement data, and stores the second measurement data in the second database table.
Optionally, when the external data file stored in XLSX format or XLSB format, that is, the second measurement data, is imported into the database, the server stores the second measurement data into the content cloud, then verifies the data downloaded from the content cloud by the ETL engine, and finally stores the data in the second database table.
Optionally, as shown in fig. 8, uploading an excel data file to a content cloud, downloading data by an EIMS-ZFLH-key service, invoking an ETL job through JDBC to perform data reading, checking and warehousing operations, and monitoring the data processing job status. After the ETL processing is successful, the data is stored in an EIMS-DW number bin GaussDB, the related log is stored in an EIMS-Kettle job log library GaussDB, and a message queue message is sent to indicate the data storage state so that the server can store the data which is failed to be stored again.
In this embodiment, the quality and usability of the second measurement data can be improved by performing extraction and check on the second measurement data through the ETL service.
In an exemplary embodiment, as shown in fig. 9, in response to invoking the ETL service, verification processing is performed on the initial second measurement data to obtain second measurement data, and the second measurement data is stored in the second database table, including steps S902 to S906. Wherein:
Step S902, calling ETL service, checking the field and the field value of the initial second measurement data, and determining the field and the field value as the second measurement data after the check is successful.
Optionally, the server invokes the ETL service through the JAVA programming language, or invokes the ETL service through the python programming language, and verifies the field and the field value of the initial second measurement data, and determines that the field and the field value are the second measurement data after the verification is successful. The calling language in the embodiments of the present application is not limited herein.
Step S904, traversing the fields of each second database table according to the fields of the second measuring and calculating data.
Optionally, the server checks the sheet table in the excel data file through Javascript, checks the fields and the field values in the sheet table, and stops the operation if the check fails; the field that is checked successfully is then ready to store a second database table. And traversing the fields of each second database table according to the fields of the second measuring and calculating data. As shown in fig. 10, the fields of the second measured data need to traverse database tables such as repayment capability status table, actual capital table, approved asset table, etc.
Step S906, after the field of the second measurement data is successfully matched with the field of each second database table, storing the second measurement data into the corresponding second database table.
Optionally, when the second measurement data field is successfully matched with a field in the second database table, displaying that the second measurement data field is successfully stored in a certain database table; and when the field matching fails, displaying that the data is failed to be stored in a database table. As shown in fig. 10, the second measurement data field traverses the database tables such as the repayment capability status table, the actual capital table, the approved asset table, and the like, matches the fields in the database tables such as the repayment capability status table, the actual capital table, and the approved asset table, and stores the second measurement data in the second database table when the matching is successful. As shown in fig. 11, after all the second measurement data are subjected to the binning operation, the server determines whether the data storage state is successful or failed through the sub job.
In this embodiment, the second measurement data is extracted and checked by the ETL service and stored in the second database table, so that the quality and availability of the second measurement data can be improved, and when the server receives the measurement instruction of the user, the second database table and the second measurement data can be selected from the data warehouse according to the measurement instruction, so as to determine the measurement result subsequently.
In an exemplary embodiment, invoking the ETL service, performing verification processing on the initial second measurement data to obtain second measurement data, and storing the second measurement data in a second database table, and further including: and returning the storage state of the second measuring and calculating data through the message queue.
Optionally, the ETL service is invoked to perform verification processing on the initial second measurement data to obtain second measurement data, and after the second measurement data is stored in the second database table, the server returns the storage state of the second measurement data, for example, storage success or storage failure, through the message of the message queue KafKa, so as to manage the second measurement data.
In this embodiment, the message queue returns to the storage state of the second measurement data, so that the abnormal state of the second measurement data can be found in time. And the management of the second measurement data is convenient.
In one exemplary embodiment of the present invention,
The method comprises the steps that a server obtains a history scheme information table, a history first database table and a history third database table corresponding to a history test scheme; generating a first intermediate mapping table based on the historical first database table and the historical scheme information table, wherein fields in the first intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table; and generating a second intermediate mapping table based on the historical third database table and the historical scheme information table, wherein fields in the second intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table.
The server responds to user input operation to acquire a first database table corresponding to the current measuring and calculating scheme; acquiring initial second measurement data and storing the initial second measurement data in cloud storage; acquiring initial second measurement data from cloud storage through ETL service; invoking ETL service, checking the field and the field value of the initial second measuring and calculating data, determining the field and the field value as the second measuring and calculating data after the checking is successful, and traversing the fields of each second database table according to the field of the second measuring and calculating data; after the fields of the second measuring and calculating data are successfully matched with the fields of the second database tables, the second measuring and calculating data are stored in the corresponding second database tables. And returning the storage state of the second measuring and calculating data through the message queue.
Responding to the measuring instruction, and acquiring a second database table corresponding to the current measuring scheme; determining a scheme information table corresponding to the current measuring scheme based on a first database table and a first intermediate mapping table corresponding to the current measuring scheme; and determining a third database table corresponding to the current measuring and calculating scheme based on the scheme information table corresponding to the current measuring and calculating scheme and the second intermediate mapping table. The third database table comprises first measurement data; the second database table includes second measurement data; determining a hash key value pair corresponding to the first measurement data based on a third database table; determining a hash key value pair corresponding to the second measuring and calculating data based on the second database table; and determining a measuring result according to the hash key value pair corresponding to the first measuring data and the hash key value pair corresponding to the second measuring data.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data processing device for realizing the above related data processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing device provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.
In one exemplary embodiment, as shown in fig. 12, there is provided a data processing apparatus including: a first acquisition module 1201, a second acquisition module 1202, a table determination module 1203, and a result determination module 1204, wherein:
The first obtaining module 1201 is configured to obtain, in response to a user input operation, a first database table corresponding to a current measurement scheme.
A second obtaining module 1202, configured to obtain a second database table corresponding to the current measurement scheme in response to the measurement instruction.
The table determining module 1203 is configured to determine a third database table of the current measurement scheme based on the first database table and the intermediate mapping table.
The result determining module 1204 is configured to determine a measurement result based on the second database table and the third database table of the current measurement scheme.
In one exemplary embodiment, a data processing apparatus further includes:
The intermediate mapping table generation module is used for acquiring a history scheme information table, a history first database table and a history third database table corresponding to the history test scheme; generating a first intermediate mapping table based on the historical first database table and the historical scheme information table, wherein fields in the first intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table; and generating a second intermediate mapping table based on the historical third database table and the historical scheme information table, wherein fields in the second intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table.
The table determination module 1203 includes:
the first determining unit is used for determining a scheme information table corresponding to the current measuring and calculating scheme based on a first database table and a first middle mapping table corresponding to the current measuring and calculating scheme.
The second determining unit is used for determining a third database table corresponding to the current measuring and calculating scheme based on the scheme information table corresponding to the current measuring and calculating scheme and the second intermediate mapping table.
In one exemplary embodiment, the third database table includes first measurement data; the second database table includes second measurement data; the result determining module 1204 includes:
The first key value pair determining unit is used for determining a key value pair corresponding to the first measuring and calculating data based on the third database table.
And the second key value pair determining unit is used for determining a key value pair corresponding to the second measuring and calculating data based on the second database table.
The result determining unit is used for determining a measuring result according to the key value pair corresponding to the first measuring and calculating data and the key value pair corresponding to the second measuring and calculating data.
In one exemplary embodiment, a data processing apparatus further includes:
And the third acquisition module is used for acquiring the initial second measurement data and storing the initial second measurement data in cloud storage.
And the fourth acquisition module is used for acquiring initial second measurement and calculation data from the cloud storage through the ETL service.
The verification storage module is used for calling the ETL service, performing verification processing on the initial second measurement data to obtain second measurement data, and storing the second measurement data in a second database table.
In one exemplary embodiment, a verification storage module includes:
The verification unit is used for calling the ETL service, verifying the field and the field value of the initial second measurement data, and determining the field and the field value as the second measurement data after the verification is successful.
The traversing unit is used for traversing the fields of each second database table according to the fields of the second measuring and calculating data.
The storage unit is used for storing the second measuring and calculating data into the corresponding second database tables after the fields of the second measuring and calculating data are successfully matched with the fields of the second database tables.
In one exemplary embodiment, a data processing apparatus further includes:
the state returning module is used for returning the storage state of the second measuring and calculating data through the message queue.
Each of the modules in the above-described data processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one exemplary embodiment, a computer device is provided, which may be a server, and the internal structure thereof may be as shown in fig. 13. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing data. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.
It will be appreciated by those skilled in the art that the structure shown in FIG. 13 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (10)
1. A method of data processing, the method comprising:
responding to input operation, and acquiring a first database table corresponding to the current measuring and calculating scheme;
responding to the measuring instruction, and acquiring a second database table corresponding to the current measuring scheme;
determining a third database table of the current measurement scheme based on the first database table and the intermediate mapping table;
and determining a measuring result based on the second database table and the third database table of the current measuring scheme.
2. The method according to claim 1, wherein the generating manner of the intermediate mapping table includes:
Acquiring a history scheme information table, a history first database table and a history third database table corresponding to a history test scheme;
generating a first intermediate mapping table based on the historical first database table and the historical scheme information table, wherein fields in the first intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table;
Generating a second intermediate mapping table based on the historical third database table and the historical scheme information table, wherein fields in the second intermediate mapping table are associated with fields in the historical first database table and fields in the historical scheme information table;
The determining a third database table of the current measurement scheme based on the first database table and the middle mapping table corresponding to the current measurement scheme includes:
determining a scheme information table corresponding to the current measuring scheme based on a first database table corresponding to the current measuring scheme and the first intermediate mapping table;
and determining a third database table corresponding to the current measuring and calculating scheme based on the scheme information table corresponding to the current measuring and calculating scheme and the second intermediate mapping table.
3. The method of claim 1, wherein the third database table comprises first measurement data; the second database table includes second measurement data; the determining a measurement result based on the second database table and the third database table of the current measurement scheme includes:
determining a key value pair corresponding to the first measurement data based on the third database table;
Determining a key value pair corresponding to the second measurement data based on the second database table;
Determining a measurement result according to the key value pair corresponding to the first measurement data and the key value pair corresponding to the second measurement data.
4. The method of claim 3, wherein before obtaining the second database table corresponding to the current measurement scheme in response to the measurement instruction, further comprises:
Acquiring initial second measurement data and storing the initial second measurement data in cloud storage;
Acquiring initial second measurement data from the cloud storage through ETL service;
Invoking the ETL service, and performing verification processing on the initial second measurement data to obtain second measurement data, and storing the second measurement data in the second database table.
5. The method of claim 4, wherein invoking the ETL service performs verification processing on the initial second measurement data to obtain second measurement data, and storing the second measurement data in the second database table, comprising:
invoking ETL service, checking the field and the field value of the initial second measurement data, and determining the field and the field value as the second measurement data after the field and the field value are checked successfully;
traversing the fields of each second database table according to the fields of the second measuring and calculating data;
After the fields of the second measuring and calculating data are successfully matched with the fields of each second database table, the second measuring and calculating data are stored in the corresponding second database tables.
6. The method of claim 4, wherein said invoking said ETL service performs verification processing on said initial second measured data to obtain second measured data, and after storing in said second database table, further comprising:
and returning the storage state of the second measuring and calculating data through the message queue.
7. A data processing apparatus, the apparatus comprising:
The first acquisition module is used for responding to the input operation and acquiring a first database table corresponding to the current measuring and calculating scheme;
The second acquisition module is used for responding to the measuring and calculating instruction and acquiring a second database table corresponding to the current measuring and calculating scheme;
The table determining module is used for determining a third database table of the current measuring and calculating scheme based on the first database table and the intermediate mapping table;
the result determining module is used for determining a measuring and calculating result based on the second database table and the third database table of the current measuring and calculating scheme.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410011349.9A CN117951135A (en) | 2024-01-04 | 2024-01-04 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410011349.9A CN117951135A (en) | 2024-01-04 | 2024-01-04 | Data processing method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117951135A true CN117951135A (en) | 2024-04-30 |
Family
ID=90795346
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410011349.9A Pending CN117951135A (en) | 2024-01-04 | 2024-01-04 | Data processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117951135A (en) |
-
2024
- 2024-01-04 CN CN202410011349.9A patent/CN117951135A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112286457B (en) | Object deduplication method and device, electronic equipment and machine-readable storage medium | |
| CN117951135A (en) | Data processing method, device, computer equipment and storage medium | |
| CN115495452A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN114528045B (en) | Plug-in operation method, device, computer equipment and storage medium | |
| CN115168409B (en) | Data query method and device for database sub-tables and computer equipment | |
| CN117033172A (en) | Test data processing method, apparatus, device, storage medium and program product | |
| CN116880927A (en) | Rule management method, device, computer equipment and storage medium | |
| CN118733535A (en) | Asynchronous file import method, device, computer equipment, readable storage medium and program product | |
| CN118113530A (en) | Chip fault positioning method, device, computer equipment and storage medium | |
| CN116910069A (en) | Database updating method, device, computer equipment and storage medium | |
| CN116225830A (en) | Service operation log reporting method and device, computer equipment and storage medium | |
| CN113987322A (en) | Index data query method and device, computer equipment and computer program product | |
| CN116546103A (en) | Processing method, device, equipment, storage medium and product of distributed service | |
| CN116483532A (en) | Scheduling processing method, device, computer equipment and storage medium | |
| CN116644444A (en) | File access method, device, computer equipment and storage medium thereof | |
| CN117951223A (en) | Data synchronization method, device, computer equipment and storage medium | |
| CN117056220A (en) | Batch job testing method and device, computer equipment and storage medium | |
| CN118606981A (en) | Data processing method and device for data access interface and computer equipment | |
| CN118096171A (en) | Data gray level verification method, device, computer equipment, medium and product | |
| CN118200314A (en) | File interaction method, device, computer equipment and storage medium | |
| CN118250070A (en) | Page login method, device, computer equipment, storage medium and program product | |
| CN118535535A (en) | File management method, apparatus, computer device, storage medium, and program product | |
| CN118467409A (en) | Data caching method, apparatus, computer device, readable storage medium and program product | |
| CN116910115A (en) | Group query method, device, computer equipment and storage medium | |
| CN117459519A (en) | Traceable file processing method, traceable file processing device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |