CN112965993B - Data processing system, method, device and storage medium - Google Patents
Data processing system, method, device and storage medium Download PDFInfo
- Publication number
- CN112965993B CN112965993B CN202110340120.6A CN202110340120A CN112965993B CN 112965993 B CN112965993 B CN 112965993B CN 202110340120 A CN202110340120 A CN 202110340120A CN 112965993 B CN112965993 B CN 112965993B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- data table
- user operation
- transaction code
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a data processing system, a method, a device and a storage medium, relating to the field of automatic programming, wherein the system comprises: the device comprises a port information configuration module, a data table management module, a data field acquisition module and a data operation execution module; the port information configuration module is used for providing an access interface for the data operation execution module; the data table management module is used for transmitting the target data table identification to the data operation execution module; the data field acquisition module is used for transmitting the target data field to the data operation execution module; and the data operation execution module is used for executing data operation according to the access interface, the target data table identifier and the target data field. The technical scheme disclosed by the embodiment of the invention realizes the data extraction and/or data embedding of the database under the service system, improves the data operation efficiency of the database and improves the utilization rate of database resources.
Description
Technical Field
Embodiments of the present invention relate to the field of automatic programming, and in particular, to a data processing system, method, apparatus, and storage medium.
Background
With the continuous progress of network technology, various websites and platforms are in the sight of people, which brings great convenience to the life of people, and the websites and platforms particularly provide business services for users through business databases and record business data.
At present, data operation is performed on a service database, for example, data extraction operation is generally performed by writing complete SQL (Structured Query Language ) sentences, respectively performing data operation on each database, further obtaining corresponding data operation results, discarding the current data operation results after completing the data operation, and re-performing the data operation process if related data needs to be obtained again.
However, such a data operation manner has low data operation efficiency, low utilization rate of database resources, and lack of effective management of various data operation results, resulting in waste of data resources.
Disclosure of Invention
The embodiment of the invention provides a data processing system, a data processing method, a data processing device and a data processing storage medium, which are used for executing a script according to data sent by a user operation terminal and executing data extraction operation and/or data embedding operation on a database under a service system.
In a first aspect, an embodiment of the present invention provides a data processing system, including: the device comprises a port information configuration module, a data table management module, a data field acquisition module and a data operation execution module; the data operation execution module is respectively connected with the port information configuration module, the data table management module and the data field acquisition module;
the port information configuration module is used for acquiring port information of a database under at least one service system, and providing an access interface for the data operation execution module according to a data execution script sent by a user operation end so that the data operation execution module is connected with the database under a target service system through the access interface; the data execution script comprises a data extraction script and/or a data embedding script;
the data table management module is used for executing a script according to the data sent by the user operation end and transmitting a matched target data table identifier to the data operation execution module;
the data field acquisition module is used for executing a script according to the data sent by the user operation end and transmitting a matched target data field to the data operation execution module;
The data operation executing module is used for executing data operation according to the access interface, the target data table identifier and the target data field; the data operation comprises a data extraction operation and/or a data embedding operation.
In a second aspect, an embodiment of the present invention provides a data extraction method, including:
the port information configuration module provides a matched access interface for the data operation execution module according to the data extraction script sent by the user operation end;
the data table management module transmits the matched target data table identification to the data operation execution module according to the data extraction script sent by the user operation end;
the data field acquisition module transmits the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end;
and the data operation execution module is used for executing data extraction operation according to the access interface, the target data table identifier and the target data field.
In a third aspect, an embodiment of the present invention provides a data embedding method, including:
the port information configuration module provides a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end;
The data table management module transmits the matched target data table identification to the data operation execution module according to the data embedded script transmitted by the user operation end;
the data field acquisition module transmits the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end;
and the data operation execution module executes data embedding operation according to the access interface, the target data table identifier and the target data field.
In a fourth aspect, an embodiment of the present invention provides a data extraction apparatus, including:
the first access interface acquisition module is integrated in the port information configuration module and is used for extracting scripts according to data sent by the user operation end and providing a matched access interface for the data operation execution module;
the first target data table identifier acquisition module is integrated in the data table management module and is used for transmitting the matched target data table identifier to the data operation execution module according to the data extraction script sent by the user operation end;
the first target data field acquisition module is integrated in the data field acquisition module and is used for transmitting the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end;
And the data extraction execution module is integrated in the data operation execution module and is used for executing data extraction operation according to the access interface, the target data table identifier and the target data field.
In a fifth aspect, an embodiment of the present invention provides a data embedding apparatus, including:
the second access interface acquisition module is integrated in the port information configuration module and is used for providing a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end;
the second target data table identifier acquisition module is integrated in the data table management module and is used for transmitting the matched target data table identifier to the data operation execution module according to the data embedded script sent by the user operation end;
the second target data field acquisition module is integrated in the data field acquisition module and is used for transmitting the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end;
and the data embedding execution module is integrated in the data operation execution module and is used for executing data embedding operation according to the access interface, the target data table identifier and the target data field.
In a sixth aspect, an embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored, where the program when executed by a processor implements the data extraction method according to any embodiment of the present invention, or implements the data embedding method according to any embodiment of the present invention.
According to the technical scheme disclosed by the embodiment of the invention, the port information configuration module analyzes the data execution script, the access interface of the database under the target service system is determined, the data table management module analyzes the data execution script to obtain the target data table identification, the data field acquisition module analyzes the data execution script to obtain the target data field, and the data operation execution module executes the data operation according to the access interface, the target data table identification and the target data field, so that the data extraction and/or the data embedding of the database under the service system are realized, the data operation efficiency of the database is improved, and the utilization rate of database resources is improved.
Drawings
FIG. 1A is a block diagram illustrating a data processing system according to a first embodiment of the present invention;
FIG. 1B is a block diagram illustrating a data processing system according to a first embodiment of the present invention;
Fig. 2 is a flowchart of a data extraction method according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data embedding method according to a third embodiment of the present invention;
fig. 4 is a block diagram of a data extraction device according to a fourth embodiment of the present invention;
fig. 5 is a block diagram of a data embedding device according to a fifth embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Example 1
FIG. 1A is a block diagram illustrating a data processing system according to a first embodiment of the present invention, including: the port information configuration module 100, the data table management module 200, the data field acquisition module 300 and the data operation execution module 400; the data operation execution module 400 is respectively connected with the port information configuration module 100, the data table management module 200 and the data field acquisition module 300; the data processing system may be integrated in a server;
The port information configuration module 100 is configured to obtain port information of a database under at least one service system, and provide an access interface for the data operation execution module 400 according to a data execution script sent by a user operation end, so that the data operation execution module 400 is connected with the database under a target service system through the access interface; the data execution script comprises a data extraction script and/or a data embedding script. Because of the diversity of service types, the network transaction platform needs a plurality of service systems to respectively process different types of services, for example, the service systems can comprise a personal service system and a company service system to respectively manage service information under personal transaction and company transaction; meanwhile, due to the continuous increase of traffic, different types of traffic data may need to be managed by multiple databases (i.e., relational databases) under one traffic system; optionally, in the embodiment of the present invention, the service type and the data type of the service system are not specifically limited. The user operation end is a user side operation platform corresponding to the data processing system, and displays a data processing interface in the form of a client or a webpage, and is used for acquiring operation information of the user side and displaying a corresponding operation result.
The port information configuration module 100 stores port information of all databases under each service system and can be connected to the databases under each service system, so that the port information configuration module 100 provides an access interface, for example, an SDK (Software Development Kit ) interface, for the operation execution module 400, and the operation execution module 400 can be connected to the respective databases through the access interface; specifically, after the port information configuration module 100 obtains the data execution script sent by the user operation end, the service system corresponding to the data execution script, that is, the target service system, is obtained by parsing, and connection with all databases under the target service system is established, and by providing an access interface, the data operation execution module 400 establishes communication connection with each database under the target service system.
The data extraction is a process of extracting data from a data source, and in the embodiment of the invention, the data source is each database under the target service system; the data extraction mode may include full extraction and incremental extraction. The data embedding is a process of embedding point information into a data source, namely embedding point codes into positions needing data acquisition in a database, and further acquiring the data in the database so as to count key behaviors of a user; the data embedding can also be applied to performance test of the service system, in particular to embedding a certain amount of data into a database in the performance test of the service system, simulating the production environment and enabling the performance test result to reflect the performance condition of the system more accurately.
The data table management module 200 is configured to transmit a matched target data table identifier to the data operation execution module 400 according to the data execution script sent by the user operation end; the data table management module 200 is configured to manage the data dictionary, the data table structure, and the association relationship between the data tables of each database, and after acquiring the data execution script sent by the user operation end, parse the data execution script to acquire a corresponding target database and a target data table under the target database according to the data dictionary and the data table structure of each database, and send the database name of the target database and the table name of the target data table to the data operation execution module 400 as a target data table identifier, so that the data operation execution module 400 acquires the data table information of the data operation to be executed.
Optionally, in this embodiment of the present invention, the data table management module 200 is specifically configured to determine a matched target data table according to a transaction code in the data execution script sent by the user operation end, and transmit a target data table identifier of the target data table to the data operation execution module. The transaction code is a transaction identifier which consists of numbers and/or letters and has a specific number of bits, and is used for representing a specific transaction type; the data table management module stores at least one transaction code, a data table association relation corresponding to the transaction code and a transaction description corresponding to the transaction code; after the data execution script sent by the user operation end is obtained, if the transaction code is obtained after analysis, the corresponding data table association relation can be determined according to the analysis transaction code, and then the target data table corresponding to the data execution script is determined. Specifically, rx= (data table 1 identifier, data table 2 identifier … … data table N identifier) (transaction code 1, transaction code 2 … … transaction code M) can be obtained by the following formula, where RX represents a matching relationship between the transaction code and the data table association relationship, and N and M are natural numbers greater than or equal to 1. For example, rx= (D1, D2) (T1, T2, T3) represents the transaction code T1, the transaction code T2, and the transaction code T3, which have matching relations with the data table D1 and the data table D2, that is, the transactions corresponding to the transaction code T1, the transaction code T2, and the transaction code T3 all require data extraction from the data table D1 and the data table D2, or embedded data.
In particular, if the target data table identifier is a blank identifier, the data operation execution module 400 is specifically configured to execute, through the access interface, a data operation according to the target data field in each data table under the target service system. If the data table management module 200 does not acquire the target data table identifier after analyzing the data execution script, that is, the blank identifier, for example, the data table management module 200 cannot acquire the matched target data table according to the acquired transaction code, the data operation execution module 400 needs to execute the data operation according to the target data field in each data table under the target service system through the above access interface, so as to ensure the integrity of the data.
Optionally, in the embodiment of the present invention, the data table management module 200 is further configured to send a transaction code list to a user operation end according to a data execution instruction sent by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code; the data execution instruction comprises a data extraction instruction and/or a data embedding instruction. The data table management module stores a defined transaction code list in advance, and a user can add a target transaction code to be acquired into the data execution script in a clicking or copying mode according to the transaction code list so as to improve the editing efficiency and the editing accuracy of the data execution script.
Optionally, in the embodiment of the present invention, the data table management module 200 is further configured to, when a transaction code adding instruction is obtained, add a new candidate transaction code, a data table association relationship corresponding to the new candidate transaction code, and a transaction description corresponding to the new candidate transaction code in the transaction code list according to the transaction code adding instruction. The data table management module 200 has an expansion function of a transaction code list, and when a transaction code adding instruction is acquired, the data table management module 200 updates the information into the transaction code list to meet the requirement of service development, and ensures that the transaction code list comprises transaction codes corresponding to newly added services.
The data field obtaining module 300 is configured to transmit a matched target data field to the data operation executing module 400 according to the data execution script sent by the user operation end; after the data field obtaining module 300 obtains the data execution script sent by the user operation end, the data field obtaining module analyzes and obtains a target data field corresponding to the data execution script, namely field information of the execution data operation in the data table, and the target data field marks the data operation position in the data table.
Optionally, in the embodiment of the present invention, the data field obtaining module 300 is further configured to send a data execution template to the user operation end according to a data execution instruction sent by the user operation end, so that the user operation end obtains a data execution script through the data execution template; the data execution template comprises a data extraction template and/or a data embedding template. After a user triggers a data execution instruction in a user operation end, the user operation end sends the data execution instruction to the data field acquisition module 300, the data field acquisition module 300 displays a data execution template through the user operation end, and a user can write a data execution script through the data execution template; after the data field obtaining module 300 obtains the data execution script after the code editing is completed, the target data field in the data execution script can be determined through the parsing process disclosed in the above technical scheme.
As shown in fig. 1B, optionally, in an embodiment of the present invention, the data processing system further includes: a result aggregation module 500; the result aggregation module 500 is configured to obtain a data extraction result after the data operation execution module 400 executes the data extraction operation, and aggregate the data extraction result. The result aggregation module 500 performs aggregation operations on the data extraction results, including data ordering and data synthesis, further optimizing the data structure of the data extraction results and reducing the amount of data.
Optionally, in an embodiment of the present invention, the data processing system further includes: a data management module 600; the data management module 600 is configured to store the aggregation result transmitted by the result aggregation module 500, and query the matched target aggregation result according to a data query instruction sent by the user operation end. The data management module 600 stores the aggregated data extraction results corresponding to each transaction, and can obtain the matched data extraction results according to the transaction code in the data query instruction, so as to multiplex the data extraction results of each transaction and avoid data resource waste.
Optionally, in the embodiment of the present invention, the data management module 600 is specifically configured to send the target aggregation result to the user operation end if the matched target aggregation result is obtained, send a prompt for no query result to the user operation end if the matched target aggregation result is not obtained, and display a data extraction key through the user operation end to guide the user to send a data extraction instruction. The data management module 600 indicates that the data information matched with the current transaction is not stored when the matched target aggregation result is not obtained according to the data query instruction, sends a prompt of no query result to the user operation terminal at this time, and guides the user to send a data extraction instruction.
Optionally, in an embodiment of the present invention, the data processing system further includes: a rights management module 700; the rights management module 700 is configured to set a user right according to a rights setting instruction sent by the user operation end. The user rights comprise administrator rights, ordinary user browsing rights, system user rights and system user group management rights; the administrator authority has the highest authority of the data processing system, can be used for setting other user authorities and has the operation authority under each menu; the common user browsing permission is a permission which is set for the common user and only has a browsing function; the system user authority can perform data operation in a database under a service system; the system user group management authority can perform data operation in a database under one service system, can manage the system user authority of each user under one service system, and sets the system user authority for each user.
According to the technical scheme disclosed by the embodiment of the invention, the port information configuration module analyzes the data execution script, the access interface of the database under the target service system is determined, the data table management module analyzes the data execution script to obtain the target data table identification, the data field acquisition module analyzes the data execution script to obtain the target data field, and the data operation execution module executes the data operation according to the access interface, the target data table identification and the target data field, so that the data extraction and/or the data embedding of the database under the service system are realized, the data operation efficiency of the database is improved, and the utilization rate of database resources is improved.
Example two
Fig. 2 is a flowchart of a data extraction method provided in a second embodiment of the present disclosure, where the embodiment is applicable to performing data extraction operations on a database under a service system according to a data extraction script sent by a user operation end, and the method may be performed by a data extraction device in a fourth embodiment of the present disclosure, where the device may be implemented by software and/or hardware and is integrated in a data processing system according to the first embodiment, and the method specifically includes the following steps:
s210, the port information configuration module provides a matched access interface for the data operation execution module according to the data extraction script sent by the user operation end.
Optionally, in an embodiment of the present invention, before the port information configuration module provides the matched access interface to the data operation execution module according to the data extraction script sent by the user operation end, the method further includes: the data table management module sends a transaction code list to the user operation end according to a data extraction instruction sent by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code.
Optionally, in an embodiment of the present invention, before the port information configuration module provides the matched access interface to the data operation execution module according to the data extraction script sent by the user operation end, the method further includes: the data field acquisition module sends a data extraction template to the user operation end according to a data extraction instruction sent by the user operation end, so that the user operation end acquires a data extraction script through the data extraction template.
S220, the data table management module transmits the matched target data table identification to the data operation execution module according to the data extraction script sent by the user operation end.
Optionally, in an embodiment of the present invention, the data table management module transmits, to a data operation execution module, a matched target data table identifier according to the data extraction script sent by the user operation end, including: and the data table management module determines a matched target data table according to the transaction code in the data extraction script sent by the user operation end, and transmits a target data table identifier of the target data table to the data operation execution module.
And S230, the data field acquisition module transmits the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end.
S240, the data operation execution module executes data extraction operation according to the access interface, the target data table identifier and the target data field.
Optionally, in an embodiment of the present invention, the data extraction method further includes: when the data table management module acquires a transaction code adding instruction, adding a new alternative transaction code, a data table association relation corresponding to the new alternative transaction code and a transaction description corresponding to the new alternative transaction code into the transaction code list according to the transaction code adding instruction.
Optionally, in an embodiment of the present invention, after the data operation execution module executes the data extraction operation according to the access interface, the target data table identifier, and the target data field, the data operation execution module further includes: and the result aggregation module acquires the data extraction result after the data operation execution module executes the data extraction operation, and aggregates the data extraction result.
Optionally, in an embodiment of the present invention, after the result aggregation module obtains a data extraction result after the data operation execution module executes the data extraction operation, and aggregates the data extraction result, the method further includes: and the data management module stores the aggregation result transmitted by the result aggregation module.
Optionally, in an embodiment of the present invention, the data extraction method further includes: the data management module inquires a matched target aggregation result according to a data inquiry instruction sent by the user operation end; if a matched target aggregation result is obtained, the target aggregation result is sent to the user operation end; and if the matched target aggregation result is not obtained, sending a prompt of no query result to the user operation end, and displaying a data extraction key through the user operation end so as to guide a user to send a data extraction instruction.
According to the technical scheme disclosed by the embodiment of the invention, the port information configuration module analyzes the data extraction script, the access interface of the database under the target service system is determined, the data table management module analyzes the data extraction script to obtain the target data table identification, the data field acquisition module analyzes the data extraction script to obtain the target data field, and the data operation execution module executes the data extraction operation according to the access interface, the target data table identification and the target data field, so that the data extraction of the database under the service system is realized, the data extraction efficiency of the database is improved, and the utilization rate of database resources is improved.
Example III
Fig. 3 is a flowchart of a data embedding method provided in a third embodiment of the present disclosure, where the embodiment may be adapted to perform a data embedding operation on a database under a service system according to a data embedding script sent by a user operation end, where the method may be performed by a data embedding device in a fifth embodiment of the present disclosure, where the device may be implemented by software and/or hardware and is integrated in a data processing system according to the first embodiment, and the method specifically includes the following steps:
and S310, the port information configuration module provides a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end.
Optionally, in the embodiment of the present invention, before the port information configuration module provides the matched access interface for the data operation execution module according to the data embedded script sent by the user operation end, the method further includes: the data table management module sends a transaction code list to the user operation end according to a data embedded instruction sent by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code.
Optionally, in the embodiment of the present invention, before the port information configuration module provides the matched access interface for the data operation execution module according to the data embedded script sent by the user operation end, the method further includes: the data field acquisition module sends a data embedding template to the user operation terminal according to a data embedding instruction sent by the user operation terminal, so that the user operation terminal acquires a data embedding script through the data embedding template.
S320, the data table management module transmits the matched target data table identification to the data operation execution module according to the data embedded script transmitted by the user operation end.
Optionally, in an embodiment of the present invention, the data table management module transmits, to the data operation execution module, a matched target data table identifier according to a data pre-embedded script sent by a user operation end, including: and the data table management module determines a matched target data table according to the transaction code in the data embedded script sent by the user operation end, and transmits a target data table identifier of the target data table to the data operation execution module.
S230, the data field acquisition module transmits the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end.
S240, the data operation execution module executes data embedding operation according to the access interface, the target data table identifier and the target data field.
Optionally, in an embodiment of the present invention, the data embedding method further includes: when the data table management module acquires a transaction code adding instruction, adding a new alternative transaction code, a data table association relation corresponding to the new alternative transaction code and a transaction description corresponding to the new alternative transaction code into the transaction code list according to the transaction code adding instruction.
According to the technical scheme disclosed by the embodiment of the invention, the port information configuration module analyzes the data embedding script, so that the access interface of the database under the target service system is determined, the data table management module analyzes the data embedding script, the target data table identification is obtained, the data field acquisition module analyzes the data embedding script, the target data field is obtained, and the data operation execution module executes the data embedding operation according to the access interface, the target data table identification and the target data field, so that the data embedding of the database under the service system is realized, the data embedding efficiency of the database is improved, and the utilization rate of database resources is improved.
Example IV
Fig. 4 is a block diagram of a data extraction device according to a fourth embodiment of the present invention, where the device specifically includes: the first access interface acquisition module 410, the first target data table identification acquisition module 420, the first target data field acquisition module 430, and the data extraction execution module 440.
The first access interface acquisition module 410 is integrated in the port information configuration module and is used for providing a matched access interface for the data operation execution module according to the data extraction script sent by the user operation end;
the first target data table identifier obtaining module 420 is integrated in the data table management module, and is configured to transmit the matched target data table identifier to the data operation execution module according to the data extraction script sent by the user operation end;
the first target data field obtaining module 430 is integrated in the data field obtaining module, and is configured to transmit the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end;
the data extraction execution module 440 is integrated in the data operation execution module, and is configured to execute a data extraction operation according to the access interface, the target data table identifier, and the target data field.
According to the technical scheme disclosed by the embodiment of the invention, the port information configuration module analyzes the data extraction script, the access interface of the database under the target service system is determined, the data table management module analyzes the data extraction script to obtain the target data table identification, the data field acquisition module analyzes the data extraction script to obtain the target data field, and the data operation execution module executes the data extraction operation according to the access interface, the target data table identification and the target data field, so that the data extraction of the database under the service system is realized, the data extraction efficiency of the database is improved, and the utilization rate of database resources is improved.
Optionally, on the basis of the above technical solution, the data extraction device further includes:
the first transaction code list acquisition module is integrated in the data list management module and is used for sending a transaction code list to the user operation end according to a data extraction instruction sent by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code.
Optionally, on the basis of the above technical solution, the data extraction device further includes:
the data extraction template acquisition module is integrated in the data field acquisition module and is used for sending a data extraction template to the user operation end according to a data extraction instruction sent by the user operation end so that the user operation end can acquire a data extraction script through the data extraction template.
Optionally, based on the above technical solution, the first target data table identifier obtaining module 420 is specifically configured to determine a matched target data table according to a transaction code in the data extraction script sent by the user operation end, and transmit a target data table identifier of the target data table to the data operation executing module.
Optionally, on the basis of the above technical solution, the data extraction device further includes:
the first transaction code management module is integrated in the data table management module and is used for adding a new alternative transaction code, a data table association relation corresponding to the new alternative transaction code and a transaction description corresponding to the new alternative transaction code into the transaction code list according to the transaction code addition instruction when the transaction code addition instruction is acquired.
Optionally, on the basis of the above technical solution, the data extraction device further includes:
and the aggregation execution module is integrated in the result aggregation module and is used for acquiring the data extraction result after the data operation execution module executes the data extraction operation and aggregating the data extraction result.
Optionally, on the basis of the above technical solution, the data extraction device further includes:
and the aggregation result storage module is integrated in the data management module and is used for storing the aggregation result transmitted by the result aggregation module.
Optionally, on the basis of the above technical solution, the data extraction device further includes:
the query execution module is integrated in the data management module and is used for querying the matched target aggregation result according to the data query instruction sent by the user operation end; if a matched target aggregation result is obtained, the target aggregation result is sent to the user operation end; and if the matched target aggregation result is not obtained, sending a prompt of no query result to the user operation end, and displaying a data extraction key through the user operation end so as to guide a user to send a data extraction instruction.
The device can execute the data extraction method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment may be referred to the data extraction method provided in any embodiment of the present invention.
Example five
Fig. 5 is a block diagram of a data embedding device according to a fifth embodiment of the present invention, where the device specifically includes: the second access interface acquisition module 510, the second target data table identifier acquisition module 520, the second target data field acquisition module 530, and the data embedding execution module 540.
The second access interface acquisition module 510 is integrated in the port information configuration module, and is used for providing a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end;
the second target data table identifier obtaining module 520 is integrated in the data table management module and is used for transmitting the matched target data table identifier to the data operation execution module according to the data embedded script sent by the user operation end;
the second target data field acquisition module 530 is integrated in the data field acquisition module and is used for transmitting the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end;
the data pre-embedding execution module 540 is integrated in the data operation execution module, and is configured to execute a data pre-embedding operation according to the access interface, the target data table identifier, and the target data field.
According to the technical scheme disclosed by the embodiment of the invention, the port information configuration module analyzes the data embedding script, so that the access interface of the database under the target service system is determined, the data table management module analyzes the data embedding script, the target data table identification is obtained, the data field acquisition module analyzes the data embedding script, the target data field is obtained, and the data operation execution module executes the data embedding operation according to the access interface, the target data table identification and the target data field, so that the data embedding of the database under the service system is realized, the data embedding efficiency of the database is improved, and the utilization rate of database resources is improved.
Optionally, on the basis of the above technical solution, the data pre-embedding device further includes:
the second transaction code list acquisition module is integrated in the data list management module and is used for transmitting a transaction code list to the user operation end according to a data embedded instruction transmitted by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code.
Optionally, on the basis of the above technical solution, the data pre-embedding device further includes:
the data embedded template acquisition module is integrated in the data field acquisition module and is used for transmitting the data embedded template to the user operation end according to the data embedded instruction transmitted by the user operation end so that the user operation end can acquire the data embedded script through the data embedded template.
Optionally, based on the above technical solution, the second target data table identifier obtaining module 520 is specifically configured to determine a matched target data table according to the transaction code in the data embedded script sent by the user operation end, and transmit the target data table identifier of the target data table to the data operation executing module.
Optionally, on the basis of the above technical solution, the data pre-embedding device further includes:
and the second transaction code management module is integrated in the data table management module and is used for adding a new alternative transaction code, a data table association relation corresponding to the new alternative transaction code and a transaction description corresponding to the new alternative transaction code into the transaction code list according to the transaction code addition instruction when the transaction code addition instruction is acquired.
The device can execute the data embedding method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details which are not described in detail in the present embodiment can be referred to the data embedding method provided in any embodiment of the present invention.
Example six
A sixth embodiment of the present invention further provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data extraction method according to any embodiment of the present invention, the method comprising:
the port information configuration module provides a matched access interface for the data operation execution module according to the data extraction script sent by the user operation end;
the data table management module transmits the matched target data table identification to the data operation execution module according to the data extraction script sent by the user operation end;
the data field acquisition module transmits the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end;
and the data operation execution module is used for executing data extraction operation according to the access interface, the target data table identifier and the target data field.
Or the method for embedding the data according to any embodiment of the invention is realized, and the method comprises the following steps:
the port information configuration module provides a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end;
the data table management module transmits the matched target data table identification to the data operation execution module according to the data embedded script transmitted by the user operation end;
the data field acquisition module transmits the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end;
and the data operation execution module executes data embedding operation according to the access interface, the target data table identifier and the target data field.
The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (17)
1. A data processing system, comprising: the device comprises a port information configuration module, a data table management module, a data field acquisition module and a data operation execution module; the data operation execution module is respectively connected with the port information configuration module, the data table management module and the data field acquisition module;
the port information configuration module is used for acquiring port information of a database under at least one service system, and providing an access interface for the data operation execution module according to a data execution script sent by a user operation end so that the data operation execution module is connected with the database under a target service system through the access interface; the data execution script comprises a data extraction script and/or a data embedding script;
The data table management module is used for executing a script according to the data sent by the user operation end and transmitting a matched target data table identifier to the data operation execution module;
the data table management module is specifically configured to determine a corresponding data table association relationship according to a transaction code in the data execution script sent by the user operation end, determine a matched target data table according to the data table association relationship, and transmit a target data table identifier of the target data table to the data operation execution module; wherein, the data table management module stores at least one transaction code, a data table association relation corresponding to the transaction code and a transaction description corresponding to the transaction code;
the data field acquisition module is used for executing a script according to the data sent by the user operation end and transmitting a matched target data field to the data operation execution module;
the data operation executing module is used for executing data operation according to the access interface, the target data table identifier and the target data field; the data operation comprises a data extraction operation and/or a data embedding operation.
2. The system of claim 1, wherein the data table management module is further configured to send a transaction code list to a user operation terminal according to a data execution instruction sent by the user operation terminal; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code; the data execution instruction comprises a data extraction instruction and/or a data embedding instruction.
3. The system of claim 2, wherein the data table management module is further configured to, when a transaction code addition instruction is acquired, add a new candidate transaction code, a data table association relationship corresponding to the new candidate transaction code, and a transaction description corresponding to the new candidate transaction code to the transaction code list according to the transaction code addition instruction.
4. The system of claim 1, wherein the data field obtaining module is further configured to send a data execution template to the user operation end according to a data execution instruction sent by the user operation end, so that the user operation end obtains a data execution script through the data execution template; the data execution template comprises a data extraction template and/or a data embedding template.
5. The system of claim 1, wherein the data processing system further comprises: a result aggregation module;
the result aggregation module is used for acquiring the data extraction result after the data operation execution module executes the data extraction operation and aggregating the data extraction result.
6. The system of claim 5, wherein the data processing system further comprises: a data management module;
The data management module is used for storing the aggregation result transmitted by the result aggregation module and inquiring the matched target aggregation result according to a data inquiry instruction sent by the user operation end.
7. The system of claim 6, wherein the data management module is specifically configured to send the target aggregation result to the user operation end if the matched target aggregation result is obtained, send a prompt for no query result to the user operation end if the matched target aggregation result is not obtained, and display a data extraction key through the user operation end to guide the user to send a data extraction instruction.
8. The system of claim 1, wherein the data processing system further comprises: a rights management module;
the right management module is used for setting the user right according to the right setting instruction sent by the user operation end.
9. A method of data extraction, comprising:
the port information configuration module provides a matched access interface for the data operation execution module according to the data extraction script sent by the user operation end;
the data table management module transmits the matched target data table identification to the data operation execution module according to the data extraction script sent by the user operation end;
The data table management module transmits the matched target data table identification to the data operation execution module according to the data extraction script sent by the user operation end, and specifically comprises the following steps:
the data table management module determines a corresponding data table association relation according to the transaction code in the data extraction script sent by the user operation end, determines a matched target data table according to the data table association relation, and transmits a target data table identifier of the target data table to the data operation execution module; wherein, the data table management module stores at least one transaction code, a data table association relation corresponding to the transaction code and a transaction description corresponding to the transaction code;
the data field acquisition module transmits the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end;
and the data operation execution module is used for executing data extraction operation according to the access interface, the target data table identifier and the target data field.
10. The method of claim 9, wherein before the port information configuration module provides the matched access interface to the data operation execution module according to the data extraction script sent by the user operation end, the method further comprises:
The data table management module sends a transaction code list to the user operation end according to a data extraction instruction sent by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code.
11. The method of claim 9, wherein before the port information configuration module provides the matched access interface to the data operation execution module according to the data extraction script sent by the user operation end, the method further comprises:
the data field acquisition module sends a data extraction template to the user operation end according to a data extraction instruction sent by the user operation end, so that the user operation end acquires a data extraction script through the data extraction template.
12. The data embedding method is characterized by comprising the following steps of:
the port information configuration module provides a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end;
the data table management module transmits the matched target data table identification to the data operation execution module according to the data embedded script transmitted by the user operation end;
The data table management module transmits the matched target data table identification to the data operation execution module according to the data embedded script sent by the user operation end, and specifically comprises the following steps:
the data table management module determines a corresponding data table association relation according to the transaction code in the data embedded script sent by the user operation end, determines a matched target data table according to the data table association relation, and transmits a target data table identifier of the target data table to the data operation execution module; wherein, the data table management module stores at least one transaction code, a data table association relation corresponding to the transaction code and a transaction description corresponding to the transaction code;
the data field acquisition module transmits the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end;
and the data operation execution module executes data embedding operation according to the access interface, the target data table identifier and the target data field.
13. The method of claim 12, wherein before the port information configuration module provides the matched access interface to the data operation execution module according to the data pre-embedded script sent by the user operation end, the method further comprises:
The data table management module sends a transaction code list to the user operation end according to a data embedded instruction sent by the user operation end; the transaction code list comprises at least one alternative transaction code, a data table association relation corresponding to the alternative transaction code and a transaction description corresponding to the alternative transaction code.
14. The method of claim 12, wherein before the port information configuration module provides the matched access interface to the data operation execution module according to the data pre-embedded script sent by the user operation end, the method further comprises:
the data field acquisition module sends a data embedding template to the user operation terminal according to a data embedding instruction sent by the user operation terminal, so that the user operation terminal acquires a data embedding script through the data embedding template.
15. A data extraction apparatus, comprising:
the first access interface acquisition module is integrated in the port information configuration module and is used for extracting scripts according to data sent by the user operation end and providing a matched access interface for the data operation execution module;
the first target data table identifier acquisition module is integrated in the data table management module and is used for transmitting the matched target data table identifier to the data operation execution module according to the data extraction script sent by the user operation end;
The first target data table identifier obtaining module is specifically configured to determine a corresponding data table association relationship according to a transaction code in the data extraction script sent by the user operation end, determine a matched target data table according to the data table association relationship, and transmit a target data table identifier of the target data table to the data operation executing module; wherein, the data table management module stores at least one transaction code, a data table association relation corresponding to the transaction code and a transaction description corresponding to the transaction code;
the first target data field acquisition module is integrated in the data field acquisition module and is used for transmitting the matched target data field to the data operation execution module according to the data extraction script sent by the user operation end;
and the data extraction execution module is integrated in the data operation execution module and is used for executing data extraction operation according to the access interface, the target data table identifier and the target data field.
16. A data embedding apparatus, comprising:
the second access interface acquisition module is integrated in the port information configuration module and is used for providing a matched access interface for the data operation execution module according to the data embedded script sent by the user operation end;
The second target data table identifier acquisition module is integrated in the data table management module and is used for transmitting the matched target data table identifier to the data operation execution module according to the data embedded script sent by the user operation end;
the second target data table identifier obtaining module is specifically configured to determine a corresponding data table association relationship according to a transaction code in the data embedded script sent by the user operation end, determine a matched target data table according to the data table association relationship, and transmit a target data table identifier of the target data table to the data operation executing module; wherein, the data table management module stores at least one transaction code, a data table association relation corresponding to the transaction code and a transaction description corresponding to the transaction code;
the second target data field acquisition module is integrated in the data field acquisition module and is used for transmitting the matched target data field to the data operation execution module according to the data embedded script sent by the user operation end;
and the data embedding execution module is integrated in the data operation execution module and is used for executing data embedding operation according to the access interface, the target data table identifier and the target data field.
17. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a data extraction method according to any of claims 9-11 or a data embedding method according to any of claims 12-14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110340120.6A CN112965993B (en) | 2021-03-30 | 2021-03-30 | Data processing system, method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110340120.6A CN112965993B (en) | 2021-03-30 | 2021-03-30 | Data processing system, method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112965993A CN112965993A (en) | 2021-06-15 |
| CN112965993B true CN112965993B (en) | 2023-06-20 |
Family
ID=76279629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110340120.6A Active CN112965993B (en) | 2021-03-30 | 2021-03-30 | Data processing system, method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112965993B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109213754A (en) * | 2018-03-29 | 2019-01-15 | 北京九章云极科技有限公司 | A kind of data processing system and data processing method |
| CN110362562A (en) * | 2019-07-16 | 2019-10-22 | 中国工商银行股份有限公司 | The method and system of big data sample drawn data |
| CN110609826A (en) * | 2019-09-20 | 2019-12-24 | 四川长虹电器股份有限公司 | Data extraction method |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
-
2021
- 2021-03-30 CN CN202110340120.6A patent/CN112965993B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109213754A (en) * | 2018-03-29 | 2019-01-15 | 北京九章云极科技有限公司 | A kind of data processing system and data processing method |
| CN110362562A (en) * | 2019-07-16 | 2019-10-22 | 中国工商银行股份有限公司 | The method and system of big data sample drawn data |
| CN110609826A (en) * | 2019-09-20 | 2019-12-24 | 四川长虹电器股份有限公司 | Data extraction method |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112965993A (en) | 2021-06-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108920659B (en) | Data processing system, data processing method thereof, and computer-readable storage medium | |
| CN108875757B (en) | Information auditing method, server and system | |
| CN103238308B (en) | The method and system of propagating source identification information | |
| CN110825767A (en) | Data query method, platform, equipment and storage medium | |
| CN111563041B (en) | Test case on-demand accurate execution method | |
| CN113377805B (en) | Data query method and device, electronic equipment and computer readable storage medium | |
| CN111460241B (en) | Data query method and device, electronic equipment and storage medium | |
| US8489631B2 (en) | Distributing a query | |
| CN111694866A (en) | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium | |
| CN104881454A (en) | Updating method and system of parameter | |
| CN115016978A (en) | Data verification method and device, computer storage medium and electronic equipment | |
| CN112988770A (en) | Method and device for updating serial number, electronic equipment and storage medium | |
| US9401846B2 (en) | Information handling system configuration identification tool and method | |
| CN110554951A (en) | Method and device for managing embedded points | |
| CN106919607B (en) | Data access method, device and system | |
| CN112905620A (en) | Data query method and device, electronic equipment and storage medium | |
| CN114625407A (en) | Method, system, equipment and storage medium for implementing AB experiment | |
| CN112860662A (en) | Data blood relationship establishing method and device, computer equipment and storage medium | |
| CN109726213B (en) | Program code conversion method, device, medium and computing equipment | |
| CN112965993B (en) | Data processing system, method, device and storage medium | |
| CN108959294A (en) | A kind of method and apparatus accessing search engine | |
| US10003492B2 (en) | Systems and methods for managing data related to network elements from multiple sources | |
| CN116450723A (en) | Data extraction method, device, computer equipment and storage medium | |
| CN108196921B (en) | Document development method and device, computer equipment and storage medium | |
| CN107977381B (en) | Data configuration method, index management method, related device and computing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |