CN107665233B - Database data processing method and device, computer equipment and storage medium - Google Patents

Database data processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN107665233B
CN107665233B CN201710608523.8A CN201710608523A CN107665233B CN 107665233 B CN107665233 B CN 107665233B CN 201710608523 A CN201710608523 A CN 201710608523A CN 107665233 B CN107665233 B CN 107665233B
Authority
CN
China
Prior art keywords
data
task
processing
data processing
database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710608523.8A
Other languages
Chinese (zh)
Other versions
CN107665233A (en
Inventor
柴春燕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OneConnect Smart Technology Co Ltd
Original Assignee
OneConnect Financial Technology Co Ltd Shanghai
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by OneConnect Financial Technology Co Ltd Shanghai filed Critical OneConnect Financial Technology Co Ltd Shanghai
Priority to CN201710608523.8A priority Critical patent/CN107665233B/en
Priority to PCT/CN2017/104253 priority patent/WO2019019361A1/en
Publication of CN107665233A publication Critical patent/CN107665233A/en
Application granted granted Critical
Publication of CN107665233B publication Critical patent/CN107665233B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24553Query execution of query operations
    • G06F16/24561Intermediate data storage techniques for performance improvement

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to a database data processing method, a database data processing device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a data processing task execution instruction; extracting a task identifier in the data processing task execution instruction; acquiring a task script corresponding to the task identifier; extracting a data processing instruction and a data acquisition condition in the task script; acquiring target data from the databases corresponding to the database types according to the data acquisition conditions; respectively creating a temporary data table according to the target data corresponding to each database type; and processing the target data in the temporary data table according to the data processing instruction. According to the database data processing method, the database data processing device, the computer equipment and the storage medium, independent operation on the target data corresponding to each database type is not needed, and the processing efficiency of the data corresponding to each database type is improved.

Description

Database data processing method and device, computer equipment and storage medium
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a database data processing method, apparatus, computer device, and storage medium.
Background
With the development of computer technology, a large amount of data needs to be processed by a computer. With the increasing demand for data processing, the data processing capability of computers is also increasing.
In the process of storing the data, the data can be stored in databases of different database types according to business requirements. Conventional data processing techniques can only process data in a database of a single database type. Making data processing inefficient when multiple database types of data are involved.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a database data processing method, an apparatus, a computer device, and a storage medium, which address the problem of low database data processing efficiency.
A database data processing method, the method comprising:
acquiring a data processing task execution instruction;
extracting a task identifier in the data processing task execution instruction;
acquiring a task script corresponding to the task identifier;
extracting a data processing instruction and a data acquisition condition in the task script;
acquiring target data from the databases corresponding to the database types according to the data acquisition conditions;
respectively creating a temporary data table according to the target data corresponding to each database type;
and processing the target data in the temporary data table according to the data processing instruction.
A database data processing apparatus, the apparatus comprising:
the instruction acquisition module is used for acquiring a data processing task execution instruction;
the identification extraction module is used for extracting the task identification in the data processing task execution instruction;
the script acquisition module is used for acquiring the task script corresponding to the task identifier;
the script analysis module is used for extracting a data processing instruction and a data acquisition condition in the task script;
the data acquisition module is used for acquiring target data from the databases corresponding to the database types according to the data acquisition conditions;
the data table creating module is used for respectively creating temporary data tables according to the target data corresponding to each database type;
and the data processing module is used for processing the target data in the temporary data table according to the data processing instruction.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
acquiring a data processing task execution instruction;
extracting a task identifier in the data processing task execution instruction;
acquiring a task script corresponding to the task identifier;
extracting a data processing instruction and a data acquisition condition in the task script;
acquiring target data from the databases corresponding to the database types according to the data acquisition conditions;
respectively creating a temporary data table according to the target data corresponding to each database type;
and processing the target data in the temporary data table according to the data processing instruction.
One or more storage media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
acquiring a data processing task execution instruction;
extracting a task identifier in the data processing task execution instruction;
acquiring a task script corresponding to the task identifier;
extracting a data processing instruction and a data acquisition condition in the task script;
acquiring target data from the databases corresponding to the database types according to the data acquisition conditions;
respectively creating a temporary data table according to the target data corresponding to each database type;
and processing the target data in the temporary data table according to the data processing instruction.
According to the database data processing method, the database data processing device, the computer equipment and the storage medium, the edited task script is obtained according to the data processing task execution instruction, the task script is analyzed to generate the data processing task, the data processing instruction and the data obtaining condition in the data processing task are extracted, and the target data are obtained from the database corresponding to each database type according to the data obtaining condition. And respectively creating temporary data tables according to the target data corresponding to each database type, and processing the target data corresponding to each database type in the temporary data tables according to the data processing instruction by establishing the temporary data tables corresponding to each database type. The target data corresponding to each database type does not need to be independently operated, and the processing efficiency of the data corresponding to each database type is improved.
Drawings
FIG. 1 is a diagram of an application environment of a database data processing method in one embodiment;
FIG. 2 is a block diagram of a server in a database data processing system in one embodiment;
FIG. 3 is a schematic flow chart diagram illustrating a database data processing method according to one embodiment;
FIG. 4 is a flowchart illustrating steps of reading task identifiers based on task dependencies, in one embodiment;
FIG. 5 is a flowchart illustrating steps for obtaining a custom function in one embodiment;
FIG. 6 is a flowchart illustrating steps for storing processing results in one embodiment;
FIG. 7 is a block diagram showing the structure of a database data processing apparatus according to an embodiment;
FIG. 8 is a block diagram of a data processing module in one embodiment;
fig. 9 is a block diagram showing the structure of a database data processing apparatus according to another embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
FIG. 1 is a diagram of an application environment of a database data processing method. Referring to fig. 1, the database data processing method is applied to a database data processing system. The database data processing system includes a terminal 110 and a server 120, wherein the terminal 110 is connected to the server 120 through a network. The terminal 110 may be a fixed terminal, which may specifically be at least one of a printer, a scanner, and a monitor, or a mobile terminal, which may specifically be at least one of a tablet, a smartphone, a personal data assistant, and a digital camera.
FIG. 2 is a diagram illustrating an internal configuration of server 120 in the database data processing system of FIG. 1, in accordance with one embodiment. As shown in fig. 2, the server 120 includes a processor, a nonvolatile storage medium, an internal memory, and a network interface, which are connected through a system bus. The processor of the server 120 is used to provide computing and control capabilities, to support the operation of the entire server 120, the memory is used to store data, code instructions, etc., and the network interface is used to communicate with the terminal 110 over a network. The memory has stored thereon at least one computer-executable instruction executable by the processor to implement the database data processing method provided in the embodiments of the present application and applied to the server 120. The Memory may include a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM). For example, in one embodiment, the memory includes a non-volatile storage medium and an internal memory; the nonvolatile storage medium stores a processing system, a database and computer-executable instructions, wherein the database comprises a plurality of databases corresponding to a plurality of database types respectively, target data are stored in the databases, and the computer-executable instructions can be executed by a processor to realize the database data processing method; the internal memory provides a cached operating environment for the processing system and computer-executable instructions in the non-volatile storage medium.
As shown in fig. 3, in an embodiment, a database data processing method is provided, which is exemplified by applying the method to the server 120 in fig. 1, and the method specifically includes the following steps:
s302, a data processing task execution instruction is obtained.
Specifically, the terminal 110 acquires a user account input by a user, generates a page data acquisition request according to the acquired user account, and sends the page data acquisition request to the server 120. After receiving the page data acquisition request sent by the terminal 110, the server 120 parses the page data acquisition request, and extracts the user account in the page data acquisition request by parsing. The server 120 performs authority verification on the extracted user account, and returns data processing page data to the terminal 110 according to the page data acquisition request when the extracted user account is verified to have the access authority to access the data processing page. After the terminal 110 processes the received data processing page data, it displays the data processing page according to the data processing page data. The terminal 110 obtains the task identifier selected by the user in the displayed data processing page, and generates a data processing task execution instruction according to the selected task identifier. The terminal 110 transmits a data processing task execution instruction to the server 120.
S304, extracting the task identification in the data processing task execution instruction.
Specifically, the server 120, upon receiving the data processing task execution instruction sent by the terminal 110, parses the data processing task execution instruction, and extracts the task identifier in the task execution instruction through parsing. The task identifier is a unique identifier of the task.
S306, acquiring a task script corresponding to the task identifier.
Specifically, the terminal 110 obtains a task identifier and a task script corresponding to the task identifier, which are input by a user in a displayed data processing page, and sends the obtained task identifier and task script to the server 120. The server 120 correspondingly stores the received task identification and the task script. The task script is a script executed when executing a data processing task. After extracting the task identifier, the server 120 queries a task script corresponding to the extracted task identifier from the stored task script, and obtains the queried task script.
S308, extracting the data processing instruction and the data acquisition condition in the task script.
Specifically, after extracting the task script, the server 120 parses the task script, and extracts the data processing instruction and the data acquisition condition from the task script by parsing. The data processing instruction is an instruction for processing data.
In one embodiment, the server 120 parses the task script, and configures the operation resources for the data processing instructions according to the operation resource configuration parameters by parsing and identifying the data processing instructions, the data acquisition conditions, and the operation resource configuration parameters in the task script.
And S310, acquiring target data from the databases corresponding to the database types according to the data acquisition conditions.
In particular, server 120 may access databases corresponding to a plurality of database types. The databases corresponding to the multiple database types may be set on the server 120, or may be set on other servers. The server 120 obtains the access data interface of the database corresponding to each database type, and obtains the target data meeting the data obtaining condition from the corresponding database through the access data interface.
In one embodiment, the server 120 extracts the database identifier in the data obtaining condition, identifies the database type corresponding to the database identifier, obtains the database access interface corresponding to the database type, and obtains the target data meeting the data obtaining condition from the database corresponding to the database identifier according to the database access interface.
In one embodiment, the server 120 extracts the database type in the data obtaining condition, obtains a database access interface of the database type and a database corresponding to the query database type, and obtains the target data meeting the data obtaining condition from the queried database according to the database access interface.
And S312, respectively creating a temporary data table according to the target data corresponding to each database type.
Specifically, after the server 120 obtains the target data corresponding to each database type, a temporary data table in a preset format is respectively created according to the target data corresponding to each database type, so as to obtain a temporary data table corresponding to each database type, where the target data is stored in the temporary data table.
In one embodiment, S312 specifically includes the following: acquiring temporary table establishing instructions corresponding to the database types; and respectively establishing a temporary data table for the target data corresponding to the corresponding database type according to the temporary table establishing instruction.
Specifically, the server 120 obtains target data corresponding to each database type, obtains temporary table creation instructions corresponding to each database type, creates a temporary data table corresponding to each database type according to the temporary table creation instructions, converts the data format of the target data corresponding to each database type into a data format matched with the temporary data table, and stores the target data after data format conversion in the corresponding temporary data table.
And S314, processing the target data in the temporary data table according to the data processing instruction.
Specifically, after creating the temporary data tables corresponding to the database types, the server 120 performs data processing on the target data in the temporary data tables corresponding to the database types according to the data processing, and obtains the processing result by processing the target data in the temporary data tables.
In this embodiment, the edited task script is acquired according to the data processing task execution instruction, the task script is analyzed to generate a data processing task, the data processing instruction and the data acquisition condition in the data processing task are extracted, and the target data is acquired from the database corresponding to each database type according to the data acquisition condition. And respectively creating temporary data tables according to the target data corresponding to each database type, and processing the target data corresponding to each database type in the temporary data tables according to the data processing instruction by establishing the temporary data tables corresponding to each database type. The target data corresponding to each database type does not need to be independently operated, and the processing efficiency of the data corresponding to each database type is improved.
As shown in fig. 4, in an embodiment, the step S306 further includes a step of reading the task identifier according to the task dependency relationship, where the step includes the following steps:
s402, reading task identifiers according to task dependency in the data processing task execution instruction.
Specifically, the data processing task execution instruction includes a plurality of task identifiers and task dependency relationships corresponding to the plurality of task identifiers. The server 120 analyzes the data processing task execution instruction, extracts the task dependency relationship and the task identifier in the data processing task execution instruction through analysis, and reads the task identifier according to the task dependency relationship.
In one embodiment, the task identification in the data processing task execution instruction is a task flow identification arranged in terms of task dependencies. The server 120 reads the task identification from the task stream identification according to the task dependency relationship.
For example, the task flow identifier in the data processing task execution instruction is "1, 2|3|4, 5", and the task flow identifier includes task identifiers of task 1, task 2, task 3, task 4, and task 5. Wherein, the ' represents that the task has dependence, the ' is executed firstly ', the ' task corresponding to the previous task identifier, the ' is executed later ', and the ' task corresponding to the next task identifier; the "|" indicates that the tasks before and after the "|" can be executed in parallel to identify the corresponding tasks. The task execution sequence of "1, 2|3|4, 5" is to execute task 1 first, then execute task 2, task 3, and task 4 in parallel after task 1 is executed, and then execute task 5 after task 2, task 3, and task 4 are executed in parallel.
S404, acquiring the task script corresponding to the read task identifier.
Specifically, after reading the task identifier from the task execution instruction according to the task dependency relationship, the server 120 queries the task script corresponding to the task identifier from the stored task scripts, and obtains the queried task script.
In this embodiment, the data processing task execution instruction includes a plurality of task identifiers and task dependencies corresponding to the plurality of task identifiers, reads the task identifiers according to the task dependencies, and obtains the task script according to the read task identifiers. The task corresponding to each task identifier is guaranteed to be executed according to the task dependency relationship, the operation steps for executing the task are simplified, the task execution efficiency is improved, a plurality of data processing tasks can be executed according to one data processing task execution instruction, and the data processing efficiency is improved.
As shown in fig. 5, in an embodiment, the step S314 specifically includes a step of obtaining a custom function, where the step specifically includes the following steps:
and S502, extracting the self-defined function identification in the data processing instruction.
Specifically, the server 120 parses the data processing instruction, and extracts the custom function identifier in the data processing instruction through parsing. The user-defined function identifier is the unique identifier of the user-defined function.
S504, acquiring a data processing function from the custom function library according to the custom function identifier.
Specifically, the terminal 110 acquires a data processing function input by the user in the data processing page, and sends the acquired data processing function to the server 120. After receiving the data processing function, the server 120 extracts the custom function identifier in the data processing function, and correspondingly stores the received data processing function and the extracted custom function identifier in the custom function library. After extracting the custom function identifier from the data processing instruction, the server 120 queries the data processing function in the custom function library according to the extracted custom function identifier, and obtains the queried data processing function.
And S506, processing the target data in the temporary data table according to the acquired data processing function.
Specifically, the server 120 processes the target data in the temporary data table according to the obtained data processing function, and processes the target data according to the data processing function to obtain a processing result. The server 120 may transmit the obtained processing result to the terminal 110, so that the terminal 110 is displayed in the data processing page.
In the embodiment, the target data in the temporary data table of each database type can be processed according to the user-defined data processing function, so that the target data can be processed more flexibly, and the accuracy and the efficiency of data processing are improved.
As shown in fig. 6, in an embodiment, after S314, a step of storing the processing result is specifically included, and the step specifically includes the following steps:
s602, a processing result storage instruction is obtained.
Specifically, the terminal 110 displays the processing result obtained by the server 120 in the data processing page, and when detecting that the result saving button in the data processing page is clicked, triggers the processing result storage instruction, and sends the triggered processing result storage instruction to the server 120. The server 120 receives the processing result storage instruction transmitted by the terminal 110.
In one embodiment, the server 120 may detect whether the data processing process is finished, and upon detecting that the data processing process is finished, trigger a processing result storing instruction.
S604, extracting the database type in the processing result storage instruction.
Specifically, the server 120 parses the processing result storage instruction, and extracts the database type in the processing result storage instruction by parsing. And the extracted database type is the database type corresponding to the database stored in the processing result.
And S606, converting the processing result obtained by processing the target data into a data format matched with the extracted database type.
Specifically, the server 120 obtains a processing result obtained by processing the target data, determines a data format matching the extracted database type, performs data format conversion on the processing result, and converts the processing result into a data format matching the extracted database type.
And S608, storing the processing result after format conversion into a database corresponding to the extracted database type.
Specifically, the server 120 queries a database corresponding to the extracted database type, extracts a data identifier of the processing result, and stores the extracted data identifier and the processing result in the queried database in a corresponding manner.
In one embodiment, the server 120 extracts the database identifier in the processing result storage instruction, and identifies the database type to which the database corresponding to the extracted database identifier belongs. The server 120 determines the data format corresponding to the identified database type, converts the data format of the processing result into the determined data format, and stores the processing result after the format conversion into the database corresponding to the extracted database identifier.
In this embodiment, the database type in the processing result storage instruction is extracted according to the processing result storage instruction, the processing result is subjected to data format conversion, it is ensured that the data format of the processing result is matched with the database type, the processing result after format conversion is stored in the database corresponding to the database type in the processing result storage instruction, and the correctness of the data format of the processing result is ensured.
In one embodiment, a user enters a user account and a user password through a browser page and a data processing application page. The terminal 110 acquires the input user account and user password, generates a page data request according to the acquired user account and user password, and the terminal 110 sends the page data request to the server 120.
After receiving the page data request, the server 120 extracts the user account and the user password in the page data request, and verifies whether the user corresponding to the user account has the access right according to the user account and the user password. After verifying that the user corresponding to the user account has the access right, the server 120 returns data processing page data to the terminal 110.
The terminal 110 displays a data processing page according to the data processing page data. The user selects a task identifier in the data processing page and clicks a data task execution button through an input device of the terminal 110. When detecting that the data task execution button in the data processing page is clicked, the terminal 110 generates a data processing task execution instruction according to the selected task identifier and the task dependency relationship corresponding to the selected task identifier, and sends the data processing task execution instruction to the server 120.
The server 120 extracts the task identifier in the data processing task execution instruction and the task dependency corresponding to the task identifier, and splits the plurality of data processing tasks according to the task dependency and the task identifier. The server 120 extracts the task script according to the task identifier in each data processing task, and extracts the data processing instruction and the data acquisition condition in the task script.
The server 120 queries the target data from the databases corresponding to the accessible database types according to the data acquisition conditions, and acquires the target data corresponding to the database types. The server 120 creates a temporary data table corresponding to the database type according to the temporary table creation instruction respectively obtained corresponding to each database type, and stores corresponding target data to the temporary data table corresponding to the database type. Temporary data tables corresponding to each database type may be previously interacted with. And the server 120 processes the target data in the temporary data table corresponding to each database type according to the data processing instruction to obtain a processing result.
The server 120 acquires at least one of a user account, data processing time, a data processing type, a database type, and data acquisition conditions logged in by the terminal 110, and generates a data processing log according to the acquired at least one of the user account, the data processing time, the data processing type, the database type, and the data acquisition conditions.
As shown in fig. 7, in one embodiment, a database data processing apparatus 700 is provided, which specifically includes the following: instruction obtaining module 702, identification extracting module 704, script obtaining module 706, script parsing module 708, data obtaining module 710, data table creating module 712, and data processing module 714.
An instruction obtaining module 702 is configured to obtain a data processing task execution instruction.
And the identifier extracting module 704 is used for extracting the task identifier in the data processing task execution instruction.
And the script obtaining module 706 is configured to obtain a task script corresponding to the task identifier.
And the script parsing module 708 is used for extracting the data processing instruction and the data acquisition condition in the task script.
And the data acquisition module 710 is configured to acquire target data from the databases corresponding to the database types according to the data acquisition conditions.
And a data table creating module 712, configured to create temporary data tables according to the target data corresponding to each database type.
And the data processing module 714 is used for processing the target data in the temporary data table according to the data processing instruction.
In one embodiment, the data table creating module 712 is further configured to obtain a temporary table creating instruction corresponding to each database type; and respectively establishing a temporary data table for the target data corresponding to the corresponding database type according to the temporary table establishing instruction.
In this embodiment, the edited task script is acquired according to the data processing task execution instruction, the task script is analyzed to generate a data processing task, the data processing instruction and the data acquisition condition in the data processing task are extracted, and the target data is acquired from the database corresponding to each database type according to the data acquisition condition. And respectively creating temporary data tables according to the target data corresponding to each database type, and processing the target data corresponding to each database type in the temporary data tables according to the data processing instruction by establishing the temporary data tables corresponding to each database type. The target data corresponding to each database type does not need to be independently operated, and the processing efficiency of the data corresponding to each database type is improved.
In one embodiment, the script obtaining module 706 is further configured to read task identifiers according to task dependencies in the data processing task execution instruction; and acquiring a task script corresponding to the read task identifier.
In this embodiment, the data processing task execution instruction includes a plurality of task identifiers and task dependencies corresponding to the plurality of task identifiers, reads the task identifiers according to the task dependencies, and obtains the task script according to the read task identifiers. The task corresponding to each task identifier is guaranteed to be executed according to the task dependency relationship, the operation steps for executing the task are simplified, the task execution efficiency is improved, a plurality of data processing tasks can be executed according to one data processing task execution instruction, and the data processing efficiency is improved.
As shown in fig. 8, in an embodiment, the data processing module 714 specifically includes the following: a function identifier extracting module 714a, a processing function obtaining module 714b and a processing result obtaining module 714 c.
And the function identifier extracting module 714a is used for extracting the custom function identifier in the data processing instruction.
And the processing function obtaining module 714b is configured to obtain the data processing function from the custom function library according to the custom function identifier.
And a processing result obtaining module 714c, configured to process the target data in the temporary data table according to the obtained data processing function.
In the embodiment, the target data in the temporary data table of each database type can be processed according to the user-defined data processing function, so that the target data can be processed more flexibly, and the accuracy and the efficiency of data processing are improved.
As shown in fig. 9, in an embodiment, the database data processing apparatus 700 further includes the following contents: an instruction fetch module 716, a type extraction module 718, a format conversion module 720, and a data storage module 722.
And an instruction obtaining module 716, configured to obtain a processing result storage instruction.
The type extraction module 718 is configured to extract a database type in the processing result storage instruction.
And a format conversion module 720, configured to convert a processing result obtained by processing the target data into a data format matched with the extracted database type.
And the data storage module 722 is configured to store the processing result after the format conversion to the database corresponding to the extracted database type.
In this embodiment, the database type in the processing result storage instruction is extracted according to the processing result storage instruction, the processing result is subjected to data format conversion, it is ensured that the data format of the processing result is matched with the database type, the processing result after format conversion is stored in the database corresponding to the database type in the processing result storage instruction, and the correctness of the data format of the processing result is ensured.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: acquiring a data processing task execution instruction; extracting a task identifier in a data processing task execution instruction; acquiring a task script corresponding to the task identifier; extracting a data processing instruction and a data acquisition condition in the task script; acquiring target data from databases corresponding to various database types according to data acquisition conditions; respectively creating a temporary data table according to the target data corresponding to each database type; and processing the target data in the temporary data table according to the data processing instruction.
In one embodiment, obtaining a task script corresponding to a task identifier includes: reading task identifiers according to task dependency in a data processing task execution instruction; and acquiring a task script corresponding to the read task identifier.
In one embodiment, creating the temporary data table according to the target data corresponding to each database type respectively includes: acquiring temporary table establishing instructions corresponding to the database types; and respectively establishing a temporary data table for the target data corresponding to the corresponding database type according to the temporary table establishing instruction.
In one embodiment, processing the target data in the temporary data table according to the data processing instruction includes: extracting a custom function identifier in the data processing instruction; acquiring a data processing function from a custom function library according to the custom function identifier; and processing the target data in the temporary data table according to the acquired data processing function.
In one embodiment, after processing the target data in the temporary data table according to the data processing instruction, the processor executes the computer program to further implement the following steps: acquiring a processing result storage instruction; extracting the database type in the processing result storage instruction; converting a processing result obtained by processing the target data into a data format matched with the extracted database type; and storing the processing result after the format conversion to a database corresponding to the extracted database type.
In this embodiment, the edited task script is acquired according to the data processing task execution instruction, the task script is analyzed to generate a data processing task, the data processing instruction and the data acquisition condition in the data processing task are extracted, and the target data is acquired from the database corresponding to each database type according to the data acquisition condition. And respectively creating temporary data tables according to the target data corresponding to each database type, and processing the target data corresponding to each database type in the temporary data tables according to the data processing instruction by establishing the temporary data tables corresponding to each database type. The target data corresponding to each database type does not need to be independently operated, and the processing efficiency of the data corresponding to each database type is improved.
One or more storage media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: acquiring a data processing task execution instruction; extracting a task identifier in a data processing task execution instruction; acquiring a task script corresponding to the task identifier; extracting a data processing instruction and a data acquisition condition in the task script; acquiring target data from databases corresponding to various database types according to data acquisition conditions; respectively creating a temporary data table according to the target data corresponding to each database type; and processing the target data in the temporary data table according to the data processing instruction.
In one embodiment, obtaining a task script corresponding to a task identifier includes: reading task identifiers according to task dependency in a data processing task execution instruction; and acquiring a task script corresponding to the read task identifier.
In one embodiment, creating the temporary data table according to the target data corresponding to each database type respectively includes: acquiring temporary table establishing instructions corresponding to the database types; and respectively establishing a temporary data table for the target data corresponding to the corresponding database type according to the temporary table establishing instruction.
In one embodiment, processing the target data in the temporary data table according to the data processing instruction includes: extracting a custom function identifier in the data processing instruction; acquiring a data processing function from a custom function library according to the custom function identifier; and processing the target data in the temporary data table according to the acquired data processing function.
In one embodiment, after processing the target data in the temporary data table according to the data processing instruction, the processor further performs the following steps: acquiring a processing result storage instruction; extracting the database type in the processing result storage instruction; converting a processing result obtained by processing the target data into a data format matched with the extracted database type; and storing the processing result after the format conversion to a database corresponding to the extracted database type.
In this embodiment, the edited task script is acquired according to the data processing task execution instruction, the task script is analyzed to generate a data processing task, the data processing instruction and the data acquisition condition in the data processing task are extracted, and the target data is acquired from the database corresponding to each database type according to the data acquisition condition. And respectively creating temporary data tables according to the target data corresponding to each database type, and processing the target data corresponding to each database type in the temporary data tables according to the data processing instruction by establishing the temporary data tables corresponding to each database type. The target data corresponding to each database type does not need to be independently operated, and the processing efficiency of the data corresponding to each database type is improved.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (9)

1. A database data processing method, the method comprising:
acquiring a data processing task execution instruction;
extracting a task identifier in the data processing task execution instruction;
acquiring a task script corresponding to the task identifier;
extracting a data processing instruction and a data acquisition condition in the task script;
acquiring target data from the databases corresponding to the database types according to the data acquisition conditions;
respectively creating a temporary data table according to the target data corresponding to each database type;
processing the target data in the temporary data table according to the data processing instruction;
the data processing task execution instruction comprises a plurality of task identifiers and task dependency relations corresponding to the task identifiers; and the task script corresponding to the task identifier is obtained by reading each task identifier according to the task dependency relationship in the data processing task execution instruction, and obtaining the task script corresponding to the read task identifier.
2. The method according to claim 1, wherein the creating a temporary data table according to the target data corresponding to each database type respectively comprises:
acquiring temporary table establishing instructions corresponding to the database types;
and respectively establishing a temporary data table for the target data corresponding to the corresponding database type according to the temporary table establishing instruction.
3. The method of claim 2, wherein the processing the target data in the temporary data table according to the data processing instruction comprises:
extracting a custom function identifier in the data processing instruction;
acquiring a data processing function from a custom function library according to the custom function identifier;
and processing the target data in the temporary data table according to the acquired data processing function.
4. The method according to any one of claims 1-3, wherein after the processing the target data in the temporary data table according to the data processing instruction, further comprising:
acquiring a processing result storage instruction;
extracting the database type in the processing result storage instruction;
converting a processing result obtained by processing the target data into a data format matched with the extracted database type;
and storing the processing result after the format conversion to a database corresponding to the extracted database type.
5. A database data processing apparatus, characterized in that the apparatus comprises:
the instruction acquisition module is used for acquiring a data processing task execution instruction;
the identification extraction module is used for extracting the task identification in the data processing task execution instruction;
the script acquisition module is used for acquiring the task script corresponding to the task identifier;
the script analysis module is used for extracting a data processing instruction and a data acquisition condition in the task script;
the data acquisition module is used for acquiring target data from the databases corresponding to the database types according to the data acquisition conditions;
the data table creating module is used for respectively creating temporary data tables according to the target data corresponding to each database type;
the data processing module is used for processing the target data in the temporary data table according to the data processing instruction;
the data processing task execution instruction comprises a plurality of task identifiers and task dependency relations corresponding to the task identifiers; the script obtaining module is further configured to read each task identifier according to the task dependency relationship in the data processing task execution instruction, and obtain a task script corresponding to the read task identifier.
6. The apparatus of claim 5, wherein the data processing module comprises:
the function identifier extraction module is used for extracting the user-defined function identifier in the data processing instruction;
the processing function acquisition module is used for acquiring a data processing function from a custom function library according to the custom function identifier;
and the processing result obtaining module is used for processing the target data in the temporary data table according to the obtained data processing function.
7. The apparatus of claim 5 or 6, further comprising:
the instruction acquisition module is used for acquiring a processing result storage instruction;
the type extraction module is used for extracting the database type in the processing result storage instruction;
the format conversion module is used for converting a processing result obtained by processing the target data into a data format matched with the extracted database type;
and the data storage module is used for storing the processing result after the format conversion to the database corresponding to the extracted database type.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of any one of claims 1 to 4 when executing the computer program.
9. One or more storage media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the method of any one of claims 1 to 4.
CN201710608523.8A 2017-07-24 2017-07-24 Database data processing method and device, computer equipment and storage medium Active CN107665233B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201710608523.8A CN107665233B (en) 2017-07-24 2017-07-24 Database data processing method and device, computer equipment and storage medium
PCT/CN2017/104253 WO2019019361A1 (en) 2017-07-24 2017-09-29 Method and apparatus for processing data of database, computer device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710608523.8A CN107665233B (en) 2017-07-24 2017-07-24 Database data processing method and device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN107665233A CN107665233A (en) 2018-02-06
CN107665233B true CN107665233B (en) 2020-07-31

Family

ID=61122580

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710608523.8A Active CN107665233B (en) 2017-07-24 2017-07-24 Database data processing method and device, computer equipment and storage medium

Country Status (2)

Country Link
CN (1) CN107665233B (en)
WO (1) WO2019019361A1 (en)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108446175A (en) * 2018-03-12 2018-08-24 平安普惠企业管理有限公司 Task processing method, device, computer equipment and storage medium
CN108446371B (en) * 2018-03-15 2020-10-27 平安科技(深圳)有限公司 Data back-leading method and device, computer equipment and storage medium
CN110309214B (en) * 2018-04-10 2023-06-23 腾讯科技(深圳)有限公司 Instruction execution method and equipment, storage medium and server thereof
CN110489213B (en) 2018-05-15 2022-04-05 华为技术有限公司 Task processing method and processing device and computer system
CN110750597A (en) * 2018-07-23 2020-02-04 北京国双科技有限公司 Data processing method and device, storage medium and processor
CN110019434A (en) * 2018-08-23 2019-07-16 深圳市信锐网科技术有限公司 Internet of things data processing method, device, platform and storage medium
CN109284097B (en) * 2018-09-07 2022-02-15 武汉轻工大学 Method, device, system and storage medium for realizing complex data analysis
CN111339170A (en) * 2018-12-19 2020-06-26 深圳市优必选科技有限公司 Data processing method and device, computer equipment and storage medium
CN110020773A (en) * 2019-01-30 2019-07-16 阿里巴巴集团控股有限公司 A kind of data processing method, equipment and computer-readable medium
CN110750308B (en) * 2019-09-11 2024-03-01 东软集团股份有限公司 Task processing method and device, storage medium and electronic equipment
CN111078733B (en) * 2019-11-26 2024-02-09 金蝶软件(中国)有限公司 Batch task processing method, device, computer equipment and storage medium
CN112988806A (en) * 2019-12-17 2021-06-18 北京沃东天骏信息技术有限公司 Data processing method and device
CN111813783B (en) * 2020-07-27 2024-03-26 南方电网数字电网研究院有限公司 Data processing method, device, computer equipment and storage medium
CN112000644B (en) * 2020-08-21 2022-08-02 北京浪潮数据技术有限公司 Data processing method and system, electronic equipment and storage medium
CN113269547B (en) * 2021-05-31 2024-05-31 中国农业银行股份有限公司 Data processing method, device, electronic equipment and storage medium
CN113778999A (en) * 2021-09-29 2021-12-10 平安资产管理有限责任公司 Data modularization processing method and device, computer equipment and storage medium
CN114238286B (en) * 2022-02-28 2022-08-05 连连(杭州)信息技术有限公司 Data warehouse data processing method and device, electronic equipment and storage medium
CN115952172B (en) * 2023-03-08 2023-05-26 畅捷通信息技术股份有限公司 Data matching method and device based on database temporary table

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101944128A (en) * 2010-09-25 2011-01-12 中兴通讯股份有限公司 Data export and import method and device
CN102346775A (en) * 2011-09-26 2012-02-08 苏州博远容天信息科技有限公司 Method for synchronizing multiple heterogeneous source databases based on log
CN105930389A (en) * 2016-04-14 2016-09-07 北京京东尚科信息技术有限公司 Method and system for transferring data
CN105989142A (en) * 2015-02-28 2016-10-05 华为技术有限公司 Data query method and device
CN106649355A (en) * 2015-10-30 2017-05-10 北京国双科技有限公司 Method and device for processing data
CN106815353A (en) * 2017-01-20 2017-06-09 星环信息科技(上海)有限公司 A kind of method and apparatus of data query

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013070873A1 (en) * 2011-11-10 2013-05-16 Treasure Data, Inc. System and method for operating a big-data platform

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101944128A (en) * 2010-09-25 2011-01-12 中兴通讯股份有限公司 Data export and import method and device
CN102346775A (en) * 2011-09-26 2012-02-08 苏州博远容天信息科技有限公司 Method for synchronizing multiple heterogeneous source databases based on log
CN105989142A (en) * 2015-02-28 2016-10-05 华为技术有限公司 Data query method and device
CN106649355A (en) * 2015-10-30 2017-05-10 北京国双科技有限公司 Method and device for processing data
CN105930389A (en) * 2016-04-14 2016-09-07 北京京东尚科信息技术有限公司 Method and system for transferring data
CN106815353A (en) * 2017-01-20 2017-06-09 星环信息科技(上海)有限公司 A kind of method and apparatus of data query

Also Published As

Publication number Publication date
WO2019019361A1 (en) 2019-01-31
CN107665233A (en) 2018-02-06

Similar Documents

Publication Publication Date Title
CN107665233B (en) Database data processing method and device, computer equipment and storage medium
CN107766190B (en) Interface testing method and device, computer equipment and storage medium
CN107844421B (en) Interface testing method and device, computer equipment and storage medium
CN108804299B (en) Application program exception handling method and device
CN108984388B (en) Method and terminal equipment for generating automatic test case
CN109951435B (en) Equipment identifier providing method and device and risk control method and device
RU2017134532A (en) SYSTEM AND METHOD FOR EXTRACTING AND JOINT USE OF CUSTOMIZED DATA ASSOCIATED WITH THE APPLICATION
CN111683066B (en) Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium
CN109558199B (en) Information pushing method, device, equipment and storage medium
WO2019019645A1 (en) Method and apparatus for executing information pushing task, and computer device and storage medium
CN111831542B (en) API application debugging method and device and storage medium
CN111338693B (en) Model construction-based target file generation method, server and storage medium
CN111144402A (en) OCR recognition accuracy calculation method, device, equipment and storage medium
CN115208951B (en) Request processing method, request processing device, electronic equipment and computer readable storage medium
CN112597020A (en) Interface testing method and device, computer equipment and storage medium
CN111679886A (en) Heterogeneous computing resource scheduling method, system, electronic device and storage medium
CN113448640A (en) WEB application configuration detection method, device, storage medium and computer equipment
CN115563600A (en) Data auditing method and device, electronic equipment and storage medium
CN107368407B (en) Information processing method and device
CN116069838A (en) Data processing method, device, computer equipment and storage medium
CN105592169A (en) Terminal identification method and terminal identification device
JP2019101889A (en) Test execution device and program
CN113051231A (en) File analysis method and device, computer equipment and storage medium
CN108268545B (en) Method and device for establishing hierarchical user label library
CN104407846B (en) Information processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20180530

Address after: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.)

Applicant after: Shenzhen one ledger Intelligent Technology Co., Ltd.

Address before: 200000 Xuhui District, Shanghai Kai Bin Road 166, 9, 10 level.

Applicant before: Shanghai Financial Technologies Ltd

TA01 Transfer of patent application right
CB02 Change of applicant information

Address after: 518000 Room 201, building A, 1 front Bay Road, Shenzhen Qianhai cooperation zone, Shenzhen, Guangdong

Applicant after: Shenzhen one ledger Intelligent Technology Co., Ltd.

Address before: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.)

Applicant before: Shenzhen one ledger Intelligent Technology Co., Ltd.

CB02 Change of applicant information
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1250801

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant