CN112434059A

CN112434059A - Data processing method, data processing device, computer equipment and storage medium

Info

Publication number: CN112434059A
Application number: CN202110102336.9A
Authority: CN
Inventors: 蒋乾; 施亚东; 冯可; 王崟垄; 赵伦
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd; Tencent Cloud Computing Beijing Co Ltd
Priority date: 2021-01-26
Filing date: 2021-01-26
Publication date: 2021-03-02
Anticipated expiration: 2041-01-26
Also published as: CN112434059B

Abstract

The application relates to a data processing method, a data processing device, computer equipment and a storage medium. The method comprises the following steps: sending a data table processing request to a remote database; when response information fed back by the remote database in response to the data sheet processing request is received, generating a data definition language statement of a local database based on the response information; creating or updating a base table of the local database through the data definition language statement; when a data processing request aiming at the base table created or updated in the local database is received, performing language conversion on a structured query language statement in the data processing request; and performing data processing on the base table created or updated in the local database through the structured query language statement after language conversion. By adopting the method, the data processing efficiency can be improved.

Description

Data processing method, data processing device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, a computer device, and a storage medium.

Background

With the continuous progress of computer technology and the advent of the era of mass data, data required to be stored and processed by modern application systems starts to present increasingly diverse and abundant forms and characteristics, and data of different types and scales are often required to be processed in one application system, such as a traditional relational database and a large data platform database.

When an application is developed, if a database based on different SQL (Structured Query Language) is to be used at the same time, a developer needs to spend a lot of time and effort to maintain two or more independent data interfaces at the same time, and different interfaces need to use SQL languages with different syntax rules. When a corresponding data interface is not developed based on a certain SQL language, a data source based on the SQL language cannot be used, and data processing cannot be performed based on the SQL language, which results in low data processing efficiency.

Disclosure of Invention

In view of the above, it is necessary to provide a data processing method, an apparatus, a computer device, and a storage medium capable of improving data processing efficiency.

A method of data processing, the method comprising:

sending a data table processing request to a remote database;

when response information fed back by the remote database in response to the data sheet processing request is received, generating a data definition language statement of a local database based on the response information;

creating or updating a base table of the local database through the data definition language statement;

when a data processing request aiming at the base table created or updated in the local database is received, performing language conversion on a structured query language statement in the data processing request;

and performing data processing on the base table created or updated in the local database through the structured query language statement after language conversion.

A data processing apparatus, the apparatus comprising:

the request sending module is used for sending a data table processing request to a remote database;

the statement generating module is used for generating a data definition language statement of a local database based on response information when the response information fed back by the remote database in response to the data table processing request is received;

the database table creating module is used for creating or updating the database table of the local database through the data definition language statement;

the language conversion module is used for performing language conversion on a structured query language statement in the data processing request when receiving the data processing request aiming at the base table created or updated in the local database;

and the base table processing module is used for carrying out data processing on the base table created or updated in the local database through the structured query language statement after language conversion.

In one embodiment, the spreadsheet processing request comprises a hypertext transfer protocol based network request; the request sending module is further configured to: sending the network request to a first server associated with the remote database, so that the first server calls an interface of the remote database based on the network request to acquire the field information of the base table and returns response information containing the field information of the base table;

the statement generation module is further configured to: extracting base table field information of the remote database from the response information; and generating a data definition language statement of the local database according to the base table field information.

In one embodiment, the data table processing request comprises a database access request; the request sending module is further configured to: sending the database access request to the remote database to search the data definition language statement of the remote database based on the database access request, and obtaining response information containing the data definition language statement;

the statement generation module is further configured to: extracting the data definition language sentence of the remote database from the response information; and performing language conversion on the data definition language statement to obtain the data definition language statement of the local database.

In one embodiment, the library table creation module is further configured to:

establishing communication connection with a second server associated with the local database;

and injecting the data definition language statement into the second server so as to run the data definition language statement in the second server, delete the base table of the local database and/or create a new base table in the local database.

In one embodiment, the apparatus further comprises a data synchronization module, wherein the data synchronization module is configured to:

acquiring data to be synchronized from the remote database through a data synchronization tool;

and writing the data to be synchronized acquired from the remote database into the library table.

In one embodiment, the language conversion module is further configured to:

determining grammatical information of a structured query language statement in the data processing request;

acquiring a target language conversion relation corresponding to the grammar information;

and modifying the grammar keywords in the structured query language sentences according to the target language conversion relation, or adding new grammar keywords at the target positions of the structured query language sentences.

In one embodiment, the apparatus further comprises a language conversion relationship establishing module, configured to:

obtaining grammar information corresponding to the remote database;

constructing a language conversion relation between the remote database and the local database based on the grammatical difference between the grammatical information corresponding to the remote database and the grammatical information corresponding to the local database;

the language conversion module is further configured to:

and acquiring a target language conversion relation corresponding to the grammar information from the language conversion relation.

In one embodiment, the apparatus further comprises a function customization module, wherein the function customization module is configured to:

determining a function called by a structured query language statement in the data processing request;

and when the local database does not have the target function matched with the function, registering the target function matched with the function.

In one embodiment, the apparatus further comprises an exception output module, wherein the exception output module is configured to:

if the generated data definition language statement of the home terminal database has grammar errors or the execution of the data definition language statement of the home terminal database is overtime, generating abnormal information;

and outputting the abnormal information to perform abnormal source tracing based on the abnormal information.

In one embodiment, the exception output module is further to:

generating a data processing log in the process of carrying out data processing on the base table created or updated in the home database;

and outputting the data processing log to extract a failure sub-log of data processing failure from the data processing log, and generating a failure analysis report according to log information in the failure sub-log.

In one embodiment, the data processing log records execution information and failure information of the language-converted structured query language statement; the execution information includes an execution result and an execution duration.

A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

sending a data table processing request to a remote database;

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

sending a data table processing request to a remote database;

A computer program comprising computer instructions stored in a computer readable storage medium from which a processor of a computer device reads the computer instructions, the processor executing the computer instructions to cause the computer device to perform the steps of:

sending a data table processing request to a remote database;

According to the data processing method, the data processing device, the computer equipment and the storage medium, the data table processing request is sent to the remote database, and the data definition language statement of the local database is generated on the basis of the received response information fed back by the remote database in response to the data table processing request; creating or updating a base table of a local database through a data definition language statement; therefore, data in the data source based on the remote database language is imported into the local database, and even if the local database does not send a corresponding data interface based on the remote database language, the data source of the remote database language can be used in the local database, so that the data processing efficiency can be improved. In addition, when a data processing request aiming at a base table created or updated in the local database is received, language conversion is carried out on the structured query language statement in the data processing request, and the base table created or updated in the local database is subjected to data processing through the structured query language statement after language conversion, so that even if the SQL language in the data processing request is different from the language of the local database, the data processing can be realized based on the data processing request, and the efficiency of data processing is further improved.

Drawings

FIG. 1 is a diagram of an application environment of a data processing method in one embodiment;

FIG. 2 is a flow diagram illustrating a data processing method according to one embodiment;

FIG. 3 is a diagram illustrating an exemplary structure of a system for updating a library table of a home database;

FIG. 4 is a flow chart illustrating a data processing method according to another embodiment;

FIG. 5 is a block diagram of a data processing system for a home database in one embodiment;

FIG. 6 is a block diagram showing the structure of a data processing apparatus according to an embodiment;

FIG. 7 is a block diagram showing the construction of a data processing apparatus according to another embodiment;

FIG. 8 is a diagram illustrating an internal structure of a computer device in one embodiment;

FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The data processing method provided by the application can be applied to the application environment shown in fig. 1. The terminal 102 communicates with the local server 104 via a network, and the local server 104 communicates with the remote server 106 via the network. The home database 110 is associated with the local server 104, and the remote database 120 is associated with the local server 104 or the remote server 106, such as the remote database 120a is associated with the local server 104 and the remote database 120b is associated with the remote server 106 in fig. 1. The server may be implemented as a stand-alone server or as a server cluster consisting of a plurality of servers.

The data processing method is performed by a computer device (the terminal 102 or the local server 104) as an example. The computer equipment transmits a data sheet processing request to a remote database through the remote server 106, and generates a data definition language statement of a local database based on response information when response information fed back by the remote database in response to the data sheet processing request is received; creating or updating a base table of a local database through a data definition language statement; when a data processing request aiming at a base table created or updated in a local database is received, performing language conversion on a structured query language statement in the data processing request; and performing data processing on the base table created or updated in the local database through the structured query language sentence after language conversion.

The home database 110 and the remote database 120 are different databases. The data storage in the database (local database or remote database) can adopt a block chain technology, and the block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer.

The block chain underlying platform can comprise processing modules such as user management, basic service, intelligent contract and operation monitoring. The user management module is responsible for identity information management of all blockchain participants, and comprises public and private key generation maintenance (account management), key management, user real identity and blockchain address corresponding relation maintenance (authority management) and the like, and under the authorization condition, the user management module supervises and audits the transaction condition of certain real identities and provides rule configuration (wind control audit) of risk control; the basic service module is deployed on all block chain node equipment and used for verifying the validity of the service request, recording the service request to storage after consensus on the valid request is completed, for a new service request, the basic service firstly performs interface adaptation analysis and authentication processing (interface adaptation), then encrypts service information (consensus management) through a consensus algorithm, transmits the service information to a shared account (network communication) completely and consistently after encryption, and performs recording and storage; the intelligent contract module is responsible for registering and issuing contracts, triggering the contracts and executing the contracts, developers can define contract logics through a certain programming language, issue the contract logics to a block chain (contract registration), call keys or other event triggering and executing according to the logics of contract clauses, complete the contract logics and simultaneously provide the function of upgrading and canceling the contracts; the operation monitoring module is mainly responsible for deployment, configuration modification, contract setting, cloud adaptation in the product release process and visual output of real-time states in product operation, such as: alarm, monitoring network conditions, monitoring node equipment health status, and the like.

The platform product service layer provides basic capability and an implementation framework of typical application, and developers can complete block chain implementation of business logic based on the basic capability and the characteristics of the superposed business. The application service layer provides the application service based on the block chain scheme for the business participants to use.

In one embodiment, as shown in fig. 2, a data processing method is provided, which is described by taking the example that the method is applied to the computer device (terminal 102 or local server 104) in fig. 1, and includes the following steps:

s202, sending a data table processing request to a remote database.

The type of the remote database can be a relational database or a data warehouse, the relational database refers to a database which adopts a relational model to organize data, the relational database stores data in the form of rows and columns so as to be convenient for users to understand, the series of rows and columns of the relational database are called tables, one group of tables form the database, the relational database is mainly used for transaction processing and mainly comprises databases such as Oracle, DB2, MySQL, Microsoft SQL Server, Microsoft Access and the like; the data warehouse, which may also be referred to as a big data platform database, is a theme-oriented, integrated, time-varying, but relatively stable data set of information itself, and is used for supporting management decision-making processes, and mainly includes AWS Redshift, greenplus, Hive, and tdw (term Distributed ware), and the like.

The remote database can be a database of a different type from the home database, for example, the remote database is a relational database MySQL, and the home database is a big data platform database Hive; the remote database may also be a different database of the same type as the home database, for example, the remote database is a large data platform database TDW, and the home database is a large data platform database Hive.

The data table processing request is used for acquiring information for generating a data definition language statement from a remote database, and specifically may be a network request based on a hypertext transfer protocol, such as an http request, where the acquired information for generating the data definition language statement is table field information of data stored in the remote database; the data table processing request may also be a database access request, and the acquired information for generating the data definition language statement is a data definition language statement based on a remote database language stored in the remote database.

Specifically, the local server has associated therewith a local database, and when a transmission condition of the data table processing request is satisfied, the computer device generates the data table processing request and transmits the generated data table processing request to the remote database. When the computer equipment is a terminal, the terminal generates a data sheet processing request through a client of the running local database, and sends the generated data sheet processing request to the remote database through the local server. When the computer equipment is a local server, the local server directly generates a data sheet processing request and sends the generated data sheet processing request to a remote database.

The sending condition of the data sheet processing request can be at least one of the occurrence of a new data source, the occurrence of new data in a historical data source and the reaching of a preset sending time. The remote database is a database corresponding to the newly added data source or the historical data source.

In one embodiment, the newly added data source is a newly added data source requiring data import into the local database, and in order to realize data import corresponding to the newly added data source into the local database, the computer device may obtain address information of a remote database corresponding to the newly added data source, generate a data table processing request according to the address information, and send the data table processing request to the remote database.

In one embodiment, the history data source is a data source that has imported data into the local database in the history period, the newly added data in the history data source is newly added data that needs to be imported into the local database, and when the newly added data in the history data source occurs, the computer device may obtain address information of a remote database corresponding to the history data source, generate a data table processing request according to the address information, and send the data table processing request to the remote database.

In one embodiment, for a historical data source, the computer device starts timing each time a data sheet processing request is sent to the remote database to which the computer device belongs, and when the preset timing reaches the preset sending time, the computer device sends the data sheet processing request to the remote database corresponding to the historical data source again.

And S204, when response information fed back by the remote database in response to the data table processing request is received, generating a data definition language statement of the local database based on the response information.

The response information carries information for generating a data definition language statement of the home database, the information for generating the data definition language statement of the home database may be base table field information of a remote database, or a data definition language statement based on a remote database language, and it is understood that types of data table processing requests are different, and information carried by the received response information for generating the data definition language statement of the home database is also different. For example, if the data table processing request sent to the remote database is a network request based on the hypertext transfer protocol, the received response information carries the field information of the database table; and if the data table processing request sent to the remote database is a database access request, the received response information carries the data definition language statement of the remote database. The language of the remote database is the language used by the remote database, for example, if the remote database is a MySQL database, the language of the remote database is MySQL.

Data Definition Language (DDL) is a Language in which a structured query Language set is responsible for Data structure Definition and database object Definition, and is composed of three grammars, namely, CREATE, ALTER and DROP, and a database Definition Language statement, namely, a DDL statement; structured Query Language (SQL), a database Query and programming Language for accessing data and querying, updating, and managing relational database systems, includes four categories of statements: data Definition Language (DDL), Data Manipulation Language (DML), Data Control Language (DCL), and Transaction Control Language (TCL).

The data definition language statement of the local database is a data definition language statement based on a local database language, the local database language is a language used by the local database, for example, the local database is a Hive database, the remote database language is a Hive language, and the corresponding data definition language statement of the local database is Hive DDL.

Specifically, the computer device sends a data table processing request to a remote database, after receiving the data table processing request, the remote database searches corresponding information for generating a data definition language statement from the remote database based on the data table processing request, generates response information according to the information for generating the data definition language statement, and returns the generated response information to the computer device, after receiving the response information fed back by the remote database in response to the data table processing request, the computer device extracts the information for generating the data definition language statement from the response information, and generates the data definition language statement of the local database based on grammar rules of the local database language and the extracted information for generating the data definition language statement.

In one embodiment, the terminal runs a library table injection application program related to the local database, the library table injection application program comprises an external data definition language statement access module and a data definition language statement generation module, when the library table injection application program receives response information fed back by a remote database in response to a data table processing request through the external data definition language statement access module, the external data definition language statement access module extracts information for generating a data definition language statement from the response information, the extracted information for generating the data definition language statement is transmitted to the data definition language statement generation module of the library table injection application program, and the data definition language statement generation module generates the data definition language statement based on grammar rules of the local database language and the extracted information for generating the data definition language statement, and generating a data definition language statement of the local database.

And S206, creating or updating the library table of the local database through the data definition language statement.

The creating of the base table of the local database refers to creating a new data table in the local database, and importing corresponding data into the created data table, and the updating of the base table of the local database refers to importing new data into the base table to be updated.

Specifically, after generating a data definition language statement of the local database based on the received response information, the computer device may implement a mode (schema) for creating or updating the local database in the local database by running the data definition language statement in the local server, and after creating or updating the schema of the local database, import the data to be imported into the corresponding library table through the data synchronization tool, thereby implementing creation or update of the library table of the local database. The schema (schema) of the database is a persistent naming set of descriptors of the database, and specifically describes the organization and structure of the database, and the schema object is included in the schema, and may be a table (table), a column (column), a data type (data type), a view (view), a stored procedure (stored procedure), a relationship (relationships), a primary key (primary key), a foreign key (foreign key), and the like. The data synchronization tool can be a DataX, and the DataX is an offline synchronization tool for heterogeneous data sources, and is used for realizing a stable and efficient data synchronization function between various heterogeneous data sources.

In one embodiment, when the computer device is a local server, the local server directly calls a corresponding interface of the local database to execute a data definition language statement of the local database after generating the data definition language statement based on the received response information, so as to update or create a library table in the local database.

In one embodiment, when the computer device is a terminal, after generating the data definition language statement of the local database based on the received response information, the terminal injects the generated data definition language statement into the local server to run the data definition language statement in the local server, so as to update or create the library table in the local database.

In one embodiment, the terminal runs a base table injection application program related to the local database, the base table injection application program further comprises a base table injection module, after the terminal generates a data definition language statement of the local database through the data definition language statement generation module of the base table injection application program, the generated data definition language statement of the local database is sent to the local server through the base table injection module, the data definition language statement is run in the local server, the base table in the local database is deleted and/or a new base table is created in the local database, and updating or creating of the base table in the local database is achieved.

Taking the home database as Hive, the remote database as a database of different type from the home database as an example, explaining the updating and creating of the base table of the local database, fig. 3 is a schematic structural diagram of a base table updating system of the local database in an embodiment, as shown in fig. 3, a terminal runs a base table injection application program related to the local database, the library table injection application program comprises a Hive DDL access module, a Hive DDL generation module and a Hive DDL injection module, the library table injection application program can acquire the information for generating the Hive DDL through the Hive DDL access module in a mode of sending an Http request to a remote database or accessing the remote database, and generates, by the Hive DDL generation module, the Hive DDL based on the information for generating the Hive DDL, the Hive DDL is then injected into the local server by the Hive DDL injection module, and the local server runs Hive DDL to update or create the library table in the local database.

S208, when a data processing request aiming at a base table created or updated in the local database is received, performing language conversion on the structured query language statement in the data processing request.

The data processing request is used for requesting to process data in the local database, and the requested processing operation comprises data query, data addition, data deletion, data modification and the like. The data processing request carries a structured query language statement, the requested data operations are different, and the corresponding structured query language statements are different. The database language on which the structured query language statement in the data processing request is based is the same as the database language used by the sender of the data processing request, that is, the database language on which the structured query language statement in the data processing request is based may be the same as or different from the language of the database at the home terminal. The sender of the data processing request may be another terminal or server, such as the remote server 106 shown in fig. 1, or the terminal 102 in fig. 1.

The language conversion of the structured query language statement in the data processing request means that when the database language on which the structured query language statement in the data processing request is based is inconsistent with the database language of the home terminal, the structured query language statement in the data processing request is converted into the structured query language statement consistent with the database language of the home terminal.

Specifically, when receiving a data table processing request for a base table created or updated in a local database, a computer device extracts a structured query language statement for data processing from the data processing request, and determines whether a database language on which the extracted structured query language statement is based is consistent with a local database language, that is, determines whether a grammar on which the extracted structured query language statement is based is consistent with a grammar of a structured query language used by the local database, and if so, performs data processing on the base table in the local database directly through the structured query language statement in the data processing request; and if the query language statements are not consistent, performing language conversion on the structured query language statements in the data processing request to obtain the converted structured query language statements so as to perform data processing on the base tables in the local database based on the converted structured query language statements. The library tables in the local database include library tables created or updated by the steps S202 to S204, and other library tables created or updated historically.

In an embodiment, the computer device is a terminal, a client of the local database is also run on the terminal 102 shown in fig. 1, when the terminal 102 receives a data processing request sent by another application program through the client of the running local database, a structured query language statement is directly extracted from the data processing request, and when a database language on which the structured query language statement is based is inconsistent with a local database language, language conversion is performed on the structured query language statement, and the converted structured query language statement is sent to the local server, so that the local server can perform data processing on a base table in the local database based on the received converted structured query language statement.

In one embodiment, the computer device is a terminal, when the terminal receives a data processing request sent by another terminal, the terminal directly extracts a structured query language statement from the data processing request, and when a database language on which the structured query language statement is based is not consistent with a database language of the local terminal, performs language conversion on the structured query language statement, and sends the converted structured query language statement to the local terminal server, so that the local terminal server can perform data processing on a base table in the local terminal database based on the received converted structured query language statement.

In one embodiment, the computer device is a terminal, and when the terminal receives a data processing request, the terminal directly extracts a structured query language statement from the data processing request and sends the extracted structured query language statement to the home terminal server, so that the home terminal server performs language conversion on the structured query language statement when the database language on which the received structured query language statement is based is not consistent with the home terminal database language.

In one embodiment, the computer device is a local server, the terminal 102 shown in fig. 1 or another server may directly send a data processing request to the local server, and the local server directly extracts a structured query language statement from the data processing request after receiving the data processing request, and performs language conversion on the structured query language statement when a database language on which the structured query language statement is based is not consistent with a database language of the local terminal, so as to obtain a converted structured query language statement consistent with the database language of the local terminal.

And S210, performing data processing on the base table created or updated in the local database through the structured query language sentence after language conversion.

Specifically, the computer device performs language conversion on the structured query language statement in the data table processing request to obtain a language-converted structured query language statement consistent with the language of the home-end database, and then executes the language-converted structured query language statement through the home-end server, thereby implementing data processing on the base table in the home-end database. The library tables in the local database include library tables created or updated by the steps S202 to S204, and other library tables created or updated historically.

In an embodiment, when the computer device is a terminal, the terminal sends the language-converted structured query language statement to the home terminal server after obtaining the language-converted structured query language statement, and the home terminal server runs the converted data definition language statement after receiving the language-converted structured query language statement, so as to implement processing of data in the home terminal database, such as data query, data addition, data deletion, data modification, and the like.

In the above embodiment, the computer device sends the data sheet processing request to the remote database, and generates the data definition language statement of the local database based on the received response information fed back by the remote database in response to the data sheet processing request; creating or updating a base table of a local database through a data definition language statement; therefore, data in the data source based on the remote database language is imported into the local database, and the data source of the remote database language can be used in the local database even if a corresponding data interface is not sent based on the remote database language, so that the data processing efficiency can be improved. In addition, when a data processing request aiming at a base table created or updated in the local database is received, language conversion is carried out on the structured query language statement in the data processing request, and the base table created or updated in the local database is subjected to data processing through the structured query language statement after language conversion, so that even if the SQL language in the data processing request is different from the language of the local database, the data processing can be realized based on the data processing request, and the efficiency of data processing is further improved.

In one embodiment, the computer device sends the data table processing request to the first server associated with the remote database as a network request based on the hypertext transfer protocol, the first server associated with the remote database calls an interface of the remote database to which the network request points after receiving the network request, the information of the database table field of the data table in the remote database is obtained from the remote database through the interface, generating response information of the network request according to the base table field information, returning the response information to the computer equipment sending the network request, extracting the base table field information of the remote database from the response information after the computer equipment receives the response information carrying the base table field information of the remote database, and generating a data definition language statement of the local database according to the extracted field information of the base table. Wherein the first server may be the local server shown in fig. 1 or a remote server.

Specifically, the network request may be an http request, the received response information is http response information, the computer device, after receiving the http response information, parses a json character string of the http response information to obtain a parsing result, extracts base table field information of the remote database from the parsing result, and generates a data definition language statement of the local database that satisfies the syntax rule of the local database language based on the syntax rule of the local database language and the extracted base table field information.

The above embodiment is described by taking an example in which the local database is a Hive data warehouse and the remote database is a TDW database. The response information is a response message corresponding to the http request sent to the remote database, and the URL corresponding to the TDW database is:

the computer device extracts a json string "" dbName "" containing library name information from the response message received by accessing the URL: "my _ internal _ db", and json character string "tableName" containing table name information is extracted: "my _ internal _ table", a json character string "colName" containing field information is extracted: "f _ date", "col t ype": "big", "colComment": "partition field", "", "colName": "f _ aaa", "col-t ype": "string", "colComent": "comment of field f _ aaa", thereby extracting the annotations of the library name 'my _ internal _ db', the table name 'my _ internal _ table', the field 'subarea field' and the field f _ aaa from the extracted json character string containing the field information of the library table, and generating a Hive DDL statement of the Hive database based on the Hive DDL grammar rule based on the Hive data warehouse as follows:

CREATE DATABASE IF NOT EXISTS my_internal db;

USE my_internal db;

DROP TABLE IF EXISTS my_internal_table;

CREATE TABLE my internal db. my_internal_table (

f_date bigint COMMENT'partition fields',

COMMENT of f _ aaa string COMMENT 'field f _ aaa').

Considering that there may NOT be any DATABASE involved in Hive, or there may already be any DATABASE involved, these 2 cases are handled by using the syntax "CREATE DATABASE IF NOT EXISTS my _ internal _ db" of Hive. The "CREATE TABLE …" statement of Hive DDL SQL is spliced by parsing the other fields of json. Because the base table field information of the remote database is updated, when the base table of the local database is updated based on the base table field information of the remote database, the original data table can be deleted, and a data table is newly established, so that the table structure information in the Hive database is updated.

In the above embodiment, the computer device sends the network request to the first server associated with the remote database, so that the first server calls the interface of the remote database based on the network request to obtain the field information of the library table, and returns the response information including the field information of the library table, thereby generating the data definition language statement of the local database based on the field information of the library table in the response information, and further implementing creation or update of the library table of the local database, that is, implementing importing the data in the data source based on the language of the remote database into the local database, even if the local database does not send the corresponding data interface based on the language of the remote database, the data source in the language of the remote database can be used in the local database, thereby improving the efficiency of data processing.

In an embodiment, when both the remote database and the home database are associated with the local server, the computer device may directly access the remote database, a data table processing request sent by the computer device to the remote database is a database access request, the remote database searches a data definition language statement of the remote database based on the received database access request, and generates a result of database access based on the searched data definition language statement, that is, response information corresponding to the database access request, after obtaining the response information including the data definition language statement of the remote database, the computer device extracts the data definition language statement of the remote database from the response information, and performs language conversion on the data definition language statement of the remote database to obtain the data definition language statement of the home database.

Specifically, after the data definition language sentence of the remote database is extracted from the response information, syntax information of the data definition language sentence of the remote database is determined, a target definition conversion relation corresponding to the syntax information of the data definition language sentence of the remote database is obtained from a definition language conversion relation corresponding to the remote database, and syntax conversion is performed on the data definition language sentence of the remote database according to the target definition conversion relation, so that the data definition language sentence of the local database based on the local database language is obtained.

The above embodiment will be described by taking an example in which the local database is a Hive data warehouse and the remote database is a MYSQL database. The response information is a response message corresponding to the database access request sent to the remote database, and the data definition language statement of the remote database extracted from the received response message by the computer device is as follows:

CREATE TABLE my_internal_db_from mysq.my_internal_table_from mysql(

'f _ date' string CommENT 'partition field',

'f aa' big C OMMENT 'field f aa' comment ') WITH SERDEPROPERTIES (' field. delim '=' |,

'serialization.format'='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'),

STORED AS INPUTFORMAT 'org. apache. hadoop. mapred. TextInputFormat'

OUTPUTFORMAT'org.apache:hadoop.hive.ql. io.HiveIgnorekeyTextOutputFormat'

LOCATION'hdfs://my_location/source/outface/appstore/my_internal_table_from _mysql/20140315/ in hive metadata_name: my_internal_table from_mysql

the computer equipment carries out language conversion on the data definition language sentences of the remote database, and the obtained data definition language sentences of the local database are as follows:

CREATE DATABASE IF NOT EXISTS my_internal db_from_mysql;

USE my_internal db_from_mysql;

DROP TABLE IF EXISTS my_internal db_from_mysql;

CREATE TABLE my_internal db_from_mysql. my_internal db_from_mysql (

f_date bigintthe COMMENT 'partition field',

COMMENT of f _ aaa string COMMENT 'field f _ aaa').

The customs processing of the data definition language statement of the computer device comprises removing all the characters of ' WITH server provider "and the following characters in the MYSQL data definition language statement, considering that there may NOT be a related DATABASE in the Hive, or there may be a related DATABASE already, and using syntax of ' CREATE DATABASE IF NOT EXISTS … … ' of the Hive to deal WITH the 2 cases.

In the above embodiment, the computer device sends the database access request to the remote database to search the data definition language statement of the remote database based on the database access request, and obtain the response information including the data definition language statement, so that the data definition language statement of the local database can be generated based on the data definition language statement of the remote database in the response information, and further creation or update of the base table of the local database is realized, that is, data in the data source based on the remote database language is imported into the local database, even if the local database does not send a corresponding data interface based on the remote database language, the data source of the remote database language can be used in the local database, and thus, the efficiency of data processing can be improved.

In one embodiment, S206 specifically includes the following steps: establishing communication connection with a second server associated with the local database; and injecting the data definition language statements into the second server to run the data definition language statements in the second server, delete the library tables of the local database and/or create new library tables in the local database.

Specifically, the second server associated with the local data may also be referred to as a local server, when the computer device is a terminal, after generating a data definition language statement of the local database based on the received response information, the terminal injects the generated data definition language statement into the second server, and the second server implements updating or creating of a library table in the local database by executing the data definition language statement, deleting a library table in the local database and/or creating a new library table in the local database.

In the above embodiment, the computer device establishes a communication connection through the second server associated with the home database; the data definition language statement is injected into the second server, so that the data definition language statement is operated in the second server, the base table of the local database is deleted, and/or a new base table is created in the local database, that is, the data in the data source based on the remote database language is imported into the local database, even if the local database does not send a corresponding data interface based on the remote database language, the data source of the remote database language can be used in the local database, and therefore, the efficiency of data processing can be improved.

In one embodiment, after creating or updating a library table of a local database through a data definition language statement, the computer device may further acquire data to be synchronized from a remote database through a data synchronization tool; and writing the data to be synchronized acquired from the remote database into a database table.

Specifically, the computer device may monitor in real time whether new data exists in a remote database corresponding to a base table of a local database through the data synchronization tool, and when new data exists, the data synchronization tool may directly acquire the new data from the remote database and transmit the new data to the local server in a concurrent manner, and the local server writes the received new data into the corresponding base table. Wherein, the newly added data is the data to be synchronized; the data synchronization tool used may be a DataX.

In the above embodiment, the computer device obtains the data to be synchronized from the remote database through the data synchronization tool; the data to be synchronized acquired from the remote database is written into the base table, so that the newly added data in the data source based on the language of the remote database is imported into the local database, the data in the local database is updated, and the data processing efficiency is improved.

In one embodiment, S208 specifically includes the following steps: determining grammatical information of a structured query language statement in the data processing request; acquiring a target language conversion relation corresponding to the grammatical information of the data processing request; and modifying the grammar keywords in the structured query language sentences according to the target language conversion relation, or adding new grammar keywords at the target positions of the structured query language sentences.

The syntax information is an organization rule of a language, for example, SELECT, DELETE, UPDATE, and INSERT statements in DML statements of SQL, and different types of databases have respective syntax rules, that is, syntax information. The target conversion relation is a structured query language statement in the data processing request, and corresponds to the conversion relation of the local database language, that is, the database languages on which the structured query language statement before and after conversion is based are different, but the semantics corresponding to the structured query language statement before and after conversion are the same. The grammar keywords are keywords that need to be modified, added, or deleted in the original structured query language statement, i.e., the structured query language statement in the received data processing request, when performing language conversion.

Specifically, after determining grammatical information of a structured query language sentence in a data processing request, the computer device obtains a target grammatical conversion relation corresponding to the grammatical information of the data processing request from a pre-stored conversion relation library, determines a grammatical keyword in an original structured query language sentence based on the target grammatical conversion relation, determines the target grammatical keyword based on the target conversion relation when the grammatical keyword is a keyword to be modified, and replaces the grammatical keyword in the original structured query language sentence with the target grammatical keyword; when the grammar keywords are keywords to be deleted, deleting the grammar keywords in the original structured query language sentences; and when the grammar keywords are the keywords to be added, determining the target positions of the grammar keywords in the original structured query language sentences based on the target conversion relation, and adding the grammar keywords at the target positions in the original structured query language sentences.

The language conversion procedure in the above embodiment is explained by taking the following 4 scenarios as an example.

1) The data processing request is structured query language statements based on TDW grammar, the local database is Hive, the computer equipment acquires the base table information in the statements after acquiring the TDW SQL statements, modifies the base table information, and uniformly converts the base table names into: name form of the table. For example, mydb, my _ table is converted to mydb.

2) The data processing request is a structured query language statement based on MySQL grammar, the local database is a Hive, the computer equipment acquires the MySQL statement, and after two MySQL clauses are operated or a union all is operated, when insert arrives at a target table, a table alias needs to be spliced outside a select clause on the outermost layer, so that successful execution in the Hive can be ensured. Specifically, if the insert clause contains a "select … union [ all ] …" clause, then outside the "select … union [ all ] …", judge whether the end of the clause is yes after the space character is removed) "ends, and if so, add the table alias; if not, the current MySQL SQL SQL is directly returned. For example,

before conversion:

INSERT INTO TABLE table_c

SELECT f_c1,

f_c2

FROM

(SELECT f_a1 AS f_c1,

f_a2 AS f_c2

FROM table_a

UNION ALL SELECT f_b1 AS f_c1,

f_b2 AS f_c2

FROM table_b)

is converted into

INSERT INTO TABLE table_c

SELECT f_c1,

f_c2

FROM

(SELECT f_a1 AS f_c1,

f_a2 AS f_c2

FROM table_a

UNION ALL SELECT f_b1 AS f_c1,

f_b2 AS f_c2

FROM table_b) ttt

In the language conversion, a table alias of "ttt" is added outside a select clause in insert.

3) The data processing request is a structured query language statement based on grammars such as TDW, MySQL or Postgre, the grammars such as TDW, MySQL or Postgre can also be called as original grammars, the local database is Hive, after the structured query language statement based on the original grammars is obtained, the computer equipment obtains reserved words in the structured query language statement and target reserved words for replacement, and replaces the reserved words in the structured query language statement by using the target reserved words. The reserved words include "table", "type", "interval", and the like. For example: the table is replaced by the UUID _ table, the type is replaced by the UUID _ type, the UUID is a 16-bit random number, the UUID used in each replacement is consistent, and the UUIDs of different batches are inconsistent.

4) The data processing request is structured query language sentences based on TDW and MySQL grammars, the grammars such as TDW and MySQL can also be called original grammars, the home database is Hive, after the structured query language sentences based on the original grammars are obtained, the computer equipment determines whether sub-queries exist in the structured query language sentences of the original grammars, and if the sub-queries exist, the table names are supplemented in the structured query language sentences of the original grammars. Wherein the process of determining whether there is a sub-query in the structured query language statement of the original grammar is as follows:

a. starting from the outermost layer of the sql statement

b. Loop checking whether each element of a certain layer is a sub-query

If not, the sub-query:

recorded as is, returned

If it is a sub-query:

checking in a loop whether there is a sub-query in the level of the sub-query statement, i.e. recursively invoking b

Recording the returned result of the sub-query analysis

Checking if the subquery has an alias, if not adding, returning

c. Outputting the recorded result

For example,

INSERT TABLE f_qz_login_d (ftime,fuin,web_cnt,mobile_cnt,cnt_vis_ed_num,inner_cnt,pc_08_cnt)

SELECT 20201127,

fuin,

sum(web_cnt),

sum(mobile_cnt),

sum(cnt_vis_ed_num),

sum(inner_cnt),

sum(pc_08_cnt)

FROM

( SELECT fuin,

sum(web_cnt) AS web_cnt,

0 AS mobile_cnt,

sum(count_vis_ed_num) AS cnt_vis_ed_num,

sum(count_vis_num_150) inner_cnt,

sum(count_vis_num_08) pc_08_cnt

FROM

( SELECT fuin,

(count_zr_num+count_kr_num) AS web_cnt,

count_vis_ed_num,

count_vis_num_08,

count_vis_num_150

FROM f_qz_visit_d

WHERE ftime = 20201127

AND fuin IS NOT NULL

AND fuin > 10000

AND count_zr_num+count_kr_num+count_vis_ed_num > 0 )

GROUP BY fuin

UNION ALL SELECT fuin,

0 AS web_cnt,

mobile_cnt,

0 AS cnt_vis_ed_num,

0 inner_cnt,

0 pc_08_cnt

FROM f_qz_mobile_login_cnt_d

WHERE ftime = 20201127)

GROUP BY fuin

to convert to:

INSERT INTO TABLE f_qz_login_d (ftime, fuin, web_cnt, mobile_cnt, cnt_vis_ed_num, inner_cnt, pc_08_cnt)

SELECT 20201127,

fuin,

sum(web_cnt),

sum(mobile_cnt),

sum(cnt_vis_ed_num),

sum(inner_cnt),

sum(pc_08_cnt)

FROM

(SELECT *

FROM

(SELECT fuin,

sum(web_cnt) AS web_cnt,

0 AS mobile_cnt,

sum(count_vis_ed_num) AS cnt_vis_ed_num,

sum(count_vis_num_150) inner_cnt,

sum(count_vis_num_08) pc_08_cnt

FROM

(SELECT fuin,

(count_zr_num+count_kr_num) AS web_cnt,

count_vis_ed_num,

count_vis_num_08,

count_vis_num_150

FROM f_qz_visit_d

WHERE ftime = 20201127

AND fuin IS NOT NULL

AND fuin > 10000

AND count_zr_num+count_kr_num+count_vis_ed_num > 0) t_alias_tt_4

GROUP BY fuin) t_alias_tt_2

UNION ALL SELECT *

FROM

(SELECT fuin,

0 AS web_cnt,

mobile_cnt,

0 AS cnt_vis_ed_num,

0 inner_cnt,

0 pc_08_cnt

FROM f_qz_mobile_login_cnt_d

WHERE ftime = 20201127) t_alias_tt_3) t_alias_tt_5

GROUP BY fuin

here, "t _ alias _ tt _ incremented number" in the code after language conversion exemplified in 4) is a table alias complemented at the time of language conversion.

In the above embodiment, the computer device determines the syntax information of the structured query language statement in the data processing request; acquiring a target language conversion relation corresponding to the grammar information; according to the method, the grammar keywords in the structured query language sentences are modified according to the target language conversion relation, or the grammar keywords are newly added at the target position of the structured query language sentences, so that the structured query language sentences in the data processing request can be converted into the structured query language sentences supported by the database at the home terminal, and the created or updated database tables in the database at the home terminal are subjected to data processing, and therefore, even if the SQL language in the data processing request is different from the language of the database at the home terminal, the data processing can be realized based on the data processing request, and the efficiency of the data processing is further improved.

In one embodiment, each language conversion relation in the pre-stored conversion relation library is created in advance, wherein the process of creating the language conversion relation in advance includes the following steps: obtaining grammar information corresponding to a remote database; and constructing a language conversion relation between the remote database and the local database based on the grammatical difference between the grammatical information corresponding to the remote database and the grammatical information corresponding to the local database.

Specifically, the computer equipment acquires first grammatical information of a first structured query language statement based on a remote database, searches a second structured query language statement matched with the semantics of the first structured query language statement and based on a local database, and determines second grammatical information of the second structured query language statement; determining a grammatical difference between the first structured query language statement and the second structured query language statement based on the first grammatical information and the second grammatical information; and establishing a language conversion relation between the remote database and the local database based on the grammar difference.

The first structured query language statement may be a DML statement of a first SQL, correspondingly, a second structured query language statement that matches the semantics of the first structured query language statement and is based on a local database language, and may be a DML statement of a second SQL, where the first SQL and the second SQL are structured query language statements based on different database languages, and the DML statement includes SELECT, DELETE, UPDATE, and INSERT statements.

The first syntax information may be a syntax rule corresponding to the first SQL DML statement, and the second syntax information may be a syntax rule of a statement corresponding to the first SQL statement in the second SQL DML statement, for example, when the first SQL DML statement is the first SELECT statement, a semantic matching with the first SQL DML statement is the second SELECT statement based on the first SQL DML statement.

The grammatical difference between the first structured query language statement and the second structured query language statement can be a grammatical keyword difference, the grammatical keyword difference including: the first structured query language statement exists and the second structured query language statement does not have a certain key, such as a table alias; or the name form of the key of the base table of the first structured query language statement is a first name form, and the name form of the base table of the second structured query language statement is a second name form, such as mydb, mytable and mydb, mytable, or the complement table name; or the reserved word keyword in the first structured query language statement is in a first form and the reserved word keyword in the second structured query language statement is in a second form, such as "table" and "UUID _ table", "type" and "UUID _ type".

In the above embodiment, the computer device obtains the syntax information corresponding to the remote database; the method comprises the steps of constructing a language conversion relation between a remote database and a local database based on the grammatical difference between grammatical information corresponding to the remote database and grammatical information corresponding to the local database, so that when a data processing request is received, language conversion is carried out on a structured query language statement in the data processing request based on the language conversion relation constructed in advance, and then data processing is carried out on a base table created or updated in the local database.

In one embodiment, the data processing method further includes the following steps: determining a function called by a structured query language statement in the data processing request; and when the local database does not have the target function matched with the function, registering the target function matched with the function.

It can be understood that different databases often use different database languages, each database language may have some unique functions, and the functions may not run directly on other databases, in order to ensure that the local database can support the invocation of various functions, the computer device determines whether there is a target function matching the function invoked by the structured query language statement in the data processing request after determining the function invoked by the structured query language statement in the data processing request, and if so, executes the structured query language statement after language conversion to perform the step of data processing on the base table in the local database; if not, registering a target function matched with the function, so that the target function can be called when the structured query language statement after language conversion is executed, and data processing of the base table in the local database is completed.

For example, if there is a big1(string str) Function in the TDW-based structured query language statement and there is no corresponding Function in Hive, the big1() Function based on Hive can be customized by developing Hive UDF (User Define Function); for example, if there is rc4(string col1, string col2) Function in a TDW-based structured query language statement, and there is no Function corresponding to Hive, it may be specifically configured to modify a Hive Server source code by developing a Hive UDF (User Define Function), and register the source code on the Hive Server by system. For example, if there is a uniq _ str (string col1, string col2) Function in a TDW-based structured query language statement, and there is no Function corresponding to Hive, it may specifically be that the Hive Server source code is modified by developing a Hive UDF (User Define Function), and the source code is registered on the Hive Server through uniq _ str ('ab, bc, ab, ae', ','), so that the Hive sql executed on the Hive Server can directly call the Function.

In the above embodiment, the computer device determines the function called by the structured query language statement in the data processing request; when the objective function matched with the function does not exist in the local database, the objective function matched with the function is registered, so that the objective function can be called through the structured query language statement after language conversion, data processing is carried out on the base table in the local database, and the efficiency of data processing is improved.

In one embodiment, after the computer device generates the data definition language statement of the local database based on the response information, if the generated data definition language statement of the local database has a syntax error or the execution of the data definition language statement of the local database is overtime, generating exception information; and outputting the exception information to perform exception tracing based on the exception information.

The abnormal information is used for recording the reason of execution failure of the data definition language statement of the local database, so that the abnormal source tracing is performed based on the abnormal information, and developers can optimize the data processing method conveniently.

Specifically, after generating the data definition language statement of the local database based on the response information, the computer device determines whether a syntax error exists in the data definition language statement generating the local database, and if so, records syntax error information in a manner of generating exception information, wherein determining whether a syntax error process exists in the data definition language statement generating the local database may be executing the generated data definition language statement of the local database, and determining whether a syntax error exists in the executing process; when the computer equipment executes the generated data definition language sentence of the local database, if the execution is not successful within the specified time, the execution overtime is recorded in a mode of generating abnormal information, and meanwhile, the communication connection between the local server and the local database established in the execution is automatically disconnected.

In the embodiment, if the generated data definition language statement of the home terminal database has a syntax error or the execution of the data definition language statement of the home terminal database is overtime, the computer device generates and outputs the exception information to perform exception tracing based on the exception information, so that a developer optimizes the data processing method, and the efficiency of subsequent data processing is improved.

In one embodiment, the computer device generates a data processing log in the process of performing data processing on a base table created or updated in a local database through a structured query language statement after language conversion; and outputting the data processing log to extract a failure sub-log of data processing failure from the data processing log, and generating a failure analysis report according to log information in the failure sub-log.

The data processing log records execution information and failure information generated when the structured query language statement after language conversion is executed; the execution information comprises an execution result and an execution duration, and the data processing log can be further divided into a success sub-log of data processing success and a failure sub-log of data processing failure based on the execution result; the failure information records the reason for the execution failure.

Specifically, after extracting the failure sub-log from the data processing log, the computer device extracts failure information from the log information in the failure log, counts the occupation ratios of various reasons of the language-converted structured query language statement execution failure based on the failure information, and outputs various failure reasons and the corresponding failure occupation ratios in the failure analysis report, so that a developer can optimize the technical problems existing in the language conversion process based on the failure analysis report, such as finding and repairing the problems in time.

The above embodiments are explained below by taking several failure sub-logs as examples:

1) the failure analysis report records log information of "org. ap. he. service. cl. high sqlencection: Error while composition status: FAILED: semiconductor exception [ Error 10009]: Line 1:318 Invalid table alias' ocg _ om _ static _ app", and based on the log information, it can be determined that the failure reason is that a data table of "ocg _ om _ static _ app" does not exist, so that a developer can adopt steps S202 to S206 of the data processing method of the present application to create the data table in the local database in advance.

2) The failure analysis report records "org. ap. ive. service. cli. high sqlexception: Error while composition status: FAILED: selected exception [ Error 10011] and update function str _ sum 2", and based on the log information, it can be determined that the failure reason is that the function "str _ sum 2" does not exist, so that the developer can register the function "str _ sum 2" in the local server in advance.

3) The failure analysis report records "org. ap. ive. service. cli. HiveSQLException: Error while composition status: FAILED: SemanticException [ Error 10014]: Line 1:394 Wrong definitions '1': no matching method for class com.tc.tdw.udf.numformat with (binary, int, int, int, int). poissible choics: _ FUNC _ (binary) _ FUNC _ (string, binary) _ FUNC _ (string < value: down >) _ FUNC _ (hash < value: down >) _ log information, based on which it can be determined that the reason is that the parameters are not of consistent type, if the parameters are of consistent type, the parameters are not passed, the parameters are of consistent type. Therefore, the developer can convert the type of the originally transmitted field from 'int' type to 'bigint' type in advance, or reload 'numformat' to support accepting the parameters of the int type.

In the embodiment, the computer device generates and outputs the data processing log in the process of performing data processing on the base table created or updated in the local database, so as to extract the failure sub-log of data processing failure from the data processing log, and generate the failure analysis report according to the log information in the failure sub-log, so that a developer can optimize the data processing method based on the failure analysis report, and the efficiency of subsequent data processing is improved.

In one embodiment, as shown in fig. 4, a data processing method is provided, which is described by taking the example that the method is applied to the computer device (terminal 102 or local server 104) in fig. 1, and includes the following steps:

s402, a network request is sent to a first server associated with the remote database, so that the first server calls an interface of the remote database based on the network request to acquire the field information of the base table, and response information containing the field information of the base table is returned.

S404, when response information fed back by the remote database in response to the network request is received, extracting the base table field information of the remote database from the response information.

And S406, generating a data definition language statement of the local database according to the field information of the base table.

S408, sending a database access request to the remote database to search the data definition language statement of the remote database based on the database access request, and obtaining the response information containing the data definition language statement.

And S410, when response information fed back by the remote database in response to the database access request is received, extracting the data definition language sentence of the remote database from the response information.

S412, performing language conversion on the data definition language statement to obtain the data definition language statement of the local database.

And S414, establishing communication connection with a second server associated with the local database.

And S416, injecting the data definition language statement into the second server so as to run the data definition language statement in the second server, delete the library table of the local database and/or create a new library table in the local database.

S418, when receiving a data processing request aiming at a base table created or updated in the local database, determining a function called by a structured query language statement in the data processing request.

And S420, registering the target function matched with the function when the target function matched with the function does not exist in the local database.

S422, determining the grammar information of the structured query language statement in the data processing request.

S424, the target language conversion relation corresponding to the grammar information is obtained.

S426, the grammar keywords in the structured query language sentence are modified according to the target language conversion relation, or the grammar keywords are newly added at the target position of the structured query language sentence.

S428, a target function is called through the structured query language statement after language conversion, and data processing is carried out on the base table created or updated in the local database.

And S430, generating and outputting a data processing log so as to extract a failure sub-log of data processing failure from the data processing log, and generating a failure analysis report according to log information in the failure sub-log.

The application also provides an application scene, and the application scene applies the data processing method. Specifically, the data processing method is applied to the application scenario as follows:

the local database is Hive, and the remote database is a database of a different type from the local database, such as a TDW data warehouse. As shown in fig. 5, the terminal runs a library table injection application program related to the local database, where the library table injection application program includes a Hive DDL access module, a Hive DDL generation module, and a Hive DDL injection module, and the library table injection application program may acquire information for generating Hive DDL by using the Hive DDL access module in a manner of sending an Http request to the remote database or accessing the remote database, generate Hive DDL based on the information for generating Hive DDL by using the Hive DDL generation module, then inject Hive DDL into the local server by using the Hive DDL injection module, and run Hive DDL by using the local server to update or create a library table in the local database. When a terminal or a local server receives a data processing request, extracting an SQL statement from the data processing request, if the SQL statement is a dialect SQL statement, converting the dialect SQL statement into a standard Hive SQL statement, executing the converted Hive SQL statement, if the SQL statement is the standard Hive SQL statement, directly executing the SQL statement, if a target function matched with a self-defined function in the SQL statement does not exist in the local server, registering the target function in the local server, and executing the standard Hive SQL statement to call the target function after converting the SQL statement into the standard Hive SQL statement. Before executing the standard Hive SQL statement, whether the base table field information related in the Hive SQL statement exists in the base table Schema of the local database or not can be checked in advance, and after the existence is determined, the standard Hive SQL statement is executed to complete data processing.

It should be understood that although the steps in the flowcharts of fig. 2 and 4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2 and 4 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.

In one embodiment, as shown in fig. 6, a data processing apparatus is provided, which may be a part of a computer device using a software module or a hardware module, or a combination of the two, and specifically includes: a request sending module 602, a statement generating module 604, a library table creating module 606, a language converting module 608 and a library table processing module 610, wherein:

a request sending module 602, configured to send a data table processing request to a remote database.

And a statement generating module 604, configured to, when response information fed back by the remote database in response to the data table processing request is received, generate a data definition language statement of the local database based on the response information.

And a library table creation module 606 for creating or updating library tables of the local database through the data definition language statements.

The language conversion module 608 is configured to, when a data processing request for a base table created or updated in the local database is received, perform language conversion on a structured query language statement in the data processing request.

The library table processing module 610 is configured to perform data processing on a library table created or updated in the home database through the structured query language statement after language conversion.

In the above embodiment, a data table processing request is sent to the remote database, and a data definition language statement of the local database is generated based on the received response information fed back by the remote database in response to the data table processing request; creating or updating a base table of a local database through a data definition language statement; therefore, data in the data source based on the remote database language is imported into the local database, and the data source of the remote database language can be used in the local database even if a corresponding data interface is not sent based on the remote database language, so that the data processing efficiency can be improved. In addition, when a data processing request aiming at a base table created or updated in the local database is received, language conversion is carried out on the structured query language statement in the data processing request, and the base table created or updated in the local database is subjected to data processing through the structured query language statement after language conversion, so that even if the SQL language in the data processing request is different from the language of the local database, the data processing can be realized based on the data processing request, and the efficiency of data processing is further improved.

In one embodiment, the spreadsheet transaction request comprises a hypertext transfer protocol based network request; a request sending module 602, further configured to: sending a network request to a first server associated with a remote database, so that the first server calls an interface of the remote database based on the network request to acquire the field information of the base table and returns response information containing the field information of the base table;

statement generation module 604, further configured to: extracting base table field information of a remote database from the response information; and generating a data definition language statement of the local database according to the field information of the base table.

In the above embodiment, the network request is sent to the first server associated with the remote database, so that the first server invokes an interface of the remote database based on the network request to obtain the field information of the library table, and returns the response information including the field information of the library table, thereby generating the data definition language statement of the local database based on the field information of the library table in the response information, and further implementing creation or update of the library table of the local database, that is, implementing importing the data in the data source based on the language of the remote database into the local database.

In one embodiment, the data table processing request comprises a database access request; a request sending module 602, further configured to: sending a database access request to a remote database to search a data definition language statement of the remote database based on the database access request and obtain response information containing the data definition language statement;

statement generation module 604, further configured to: extracting data definition language sentences of a remote database from the response information; and performing language conversion on the data definition language statements to obtain the data definition language statements of the local database.

In the above embodiment, the database access request is sent to the remote database, so as to search the data definition language statement of the remote database based on the database access request, and obtain the response information including the data definition language statement, so that the data definition language statement of the local database can be generated based on the data definition language statement of the remote database in the response information, and further creation or update of the base table of the local database is realized, that is, data in the data source based on the remote database language is imported into the local database.

In one embodiment, the library table creation module 606 is further configured to:

and injecting the data definition language statements into the second server to run the data definition language statements in the second server, delete the library tables of the local database and/or create new library tables in the local database.

In the above embodiment, the communication connection is established through the second server associated with the home database; the data definition language statement is injected into the second server, so that the data definition language statement is operated in the second server, the base table of the local database is deleted, and/or a new base table is created in the local database, that is, the data in the data source based on the remote database language is imported into the local database, even if the local database does not send a corresponding data interface based on the remote database language, the data source of the remote database language can be used in the local database, and therefore, the efficiency of data processing can be improved.

In one embodiment, as shown in fig. 7, the apparatus further comprises a data synchronization module 612, wherein the data synchronization module 612 is configured to:

acquiring data to be synchronized from a remote database through a data synchronization tool;

and writing the data to be synchronized acquired from the remote database into a database table.

In the embodiment, the data to be synchronized is acquired from the remote database through the data synchronization tool; the data to be synchronized acquired from the remote database is written into the base table, so that the newly added data in the data source based on the language of the remote database is imported into the local database, the data in the local database is updated, and the data processing efficiency is improved.

In one embodiment, the language conversion module 608 is further configured to:

In the above embodiment, the syntax information of the structured query language statement in the data processing request is determined; acquiring a target language conversion relation corresponding to the grammar information; according to the method, the grammar keywords in the structured query language sentences are modified according to the target language conversion relation, or the grammar keywords are newly added at the target position of the structured query language sentences, so that the structured query language sentences in the data processing request can be converted into the structured query language sentences supported by the database at the home terminal, and the created or updated database tables in the database at the home terminal are subjected to data processing, and therefore, even if the SQL language in the data processing request is different from the language of the database at the home terminal, the data processing can be realized based on the data processing request, and the efficiency of the data processing is further improved.

In one embodiment, as shown in fig. 7, the apparatus further comprises a language conversion relation establishing module 614, the language conversion relation establishing module 614 is configured to:

obtaining grammar information corresponding to a remote database;

language conversion module 608, further for:

In the above embodiment, by acquiring syntax information corresponding to the remote database; the method comprises the steps of constructing a language conversion relation between a remote database and a local database based on the grammatical difference between grammatical information corresponding to the remote database and grammatical information corresponding to the local database, so that when a data processing request is received, language conversion is carried out on a structured query language statement in the data processing request based on the language conversion relation constructed in advance, and then data processing is carried out on a base table created or updated in the local database.

In one embodiment, as shown in fig. 7, the apparatus further comprises a function customization module 616, wherein the function customization module 616 is configured to:

In the above embodiment, the function called by the structured query language statement in the data processing request is determined; when the objective function matched with the function does not exist in the local database, the objective function matched with the function is registered, so that the objective function can be called through the structured query language statement after language conversion, data processing is carried out on the base table in the local database, and the efficiency of data processing is improved.

In one embodiment, as shown in fig. 7, the apparatus further comprises an exception output module 618, wherein the exception output module 618 is configured to:

if the generated data definition language statement of the local database has grammar errors or the execution of the data definition language statement of the local database is overtime, generating abnormal information;

and outputting the exception information to perform exception tracing based on the exception information.

In the embodiment, when a syntax error exists in the generated data definition language statement of the home database or the execution of the data definition language statement of the home database is overtime, exception information is generated and output to perform exception tracing based on the exception information, so that a developer optimizes the data processing method, and the efficiency of subsequent data processing is improved.

In one embodiment, the exception output module 618 is further to:

generating a data processing log in the process of carrying out data processing on a base table created or updated in a local database;

In one embodiment, the data processing log records execution information and failure information of the structured query language statement after language conversion; the execution information includes an execution result and an execution time length.

In the embodiment, in the process of performing data processing on the base table created or updated in the local database, the data processing log is generated and output, so that the failure sub-log of data processing failure is extracted from the data processing log, and the failure analysis report is generated according to the log information in the failure sub-log, so that a developer can optimize the data processing method based on the failure analysis report, and the efficiency of subsequent data processing is improved.

For specific limitations of the data processing apparatus, reference may be made to the above limitations of the data processing method, which are not described herein again. The various modules in the data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing various service data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the configurations shown in fig. 8 or 9 are merely block diagrams of some configurations relevant to the present disclosure, and do not constitute a limitation on the computing devices to which the present disclosure may be applied, and that a particular computing device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In one embodiment, a computer program product or computer program is provided that includes computer instructions stored in a computer-readable storage medium. The computer instructions are read by a processor of a computer device from a computer-readable storage medium, and the computer instructions are executed by the processor to cause the computer device to perform the steps in the above-mentioned method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method of data processing, the method comprising:

sending a data table processing request to a remote database;

2. The method of claim 1, wherein the spreadsheet processing request comprises a hypertext transfer protocol based network request; the sending of the data table processing request to the remote database includes:

sending the network request to a first server associated with the remote database, so that the first server calls an interface of the remote database based on the network request to acquire the field information of the base table and returns response information containing the field information of the base table;

the generating of the data definition language statement of the local database based on the response information comprises:

extracting base table field information of the remote database from the response information;

and generating a data definition language statement of the local database according to the base table field information.

3. The method of claim 1, wherein the data table processing request comprises a database access request; the sending of the data table processing request to the remote database includes:

sending the database access request to the remote database to search the data definition language statement of the remote database based on the database access request, and obtaining response information containing the data definition language statement;

extracting the data definition language sentence of the remote database from the response information;

and performing language conversion on the data definition language statement to obtain the data definition language statement of the local database.

4. The method of claim 1, wherein creating or updating the library tables of the home database via the data definition language statements comprises:

5. The method of claim 1, wherein after creating or updating a library table of the home database via the data definition language statement, the method further comprises:

6. The method of claim 1, wherein the language converting the structured query language statement in the data processing request comprises:

7. The method of claim 6, further comprising:

obtaining grammar information corresponding to the remote database;

the obtaining of the target language conversion relation corresponding to the grammar information includes:

8. The method according to any one of claims 1 to 7, further comprising:

9. The method according to any one of claims 1 to 7, further comprising:

10. The method according to any one of claims 1 to 7, further comprising:

11. The method of claim 10, wherein the data processing log records execution information and failure information of the language-converted structured query language statement; the execution information includes an execution result and an execution duration.

12. A data processing apparatus, characterized in that the apparatus comprises:

13. The apparatus of claim 12, wherein the request sending module is further configured to: sending the network request to a first server associated with the remote database, so that the first server calls an interface of the remote database based on the network request to acquire the field information of the base table and returns response information containing the field information of the base table;

14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 11.