CN116266185A

CN116266185A - Data processing method and device

Info

Publication number: CN116266185A
Application number: CN202111540679.XA
Authority: CN
Inventors: 杨卓群; 朱文祥; 郭星强; 关健
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-12-16
Filing date: 2021-12-16
Publication date: 2023-06-20

Abstract

The invention discloses a data processing method and device. The method comprises the following steps: receiving a data operation language (DML) request, wherein the DML request is used for requesting to operate on target data, and is irrelevant to a database type; generating a first transaction request for an OLTP database management system according to the DML request; sending the first transaction request to the OLTP database management system; acquiring a transaction log generated by the OLTP database management system in response to the first transaction request; generating a second transaction request for the OLAP database management system according to the transaction log; and sending the second transaction request to the OLAP database management system.

Description

Data processing method and device

Technical Field

The present disclosure relates to the field of databases, and in particular, to a data processing method and apparatus.

Background

Some Software application services are currently available in a wide range of applications, such as Software-as-a-Service (SaaS). These software application services often deploy both online transaction (On-Line Transaction Processing, OLTP) and online analytics (On-Line Analytical Processing, OLAP) databases to meet overall business needs.

However, although the two databases are deployed on the software application service at the same time, the specific application modes of the two databases are independent of each other, synchronization needs to be realized through independent operation, and effective fusion on the service cannot be realized, so that the use efficiency of the databases is low at present, which is a problem to be solved urgently.

Disclosure of Invention

The application provides a data processing method and device, which are used for realizing effective business fusion of an OLTP database and an OLAP database and improving the use efficiency of the current database.

In a first aspect, an embodiment of the present application provides a data processing method, which may be applied to a data processing apparatus provided in the third aspect of the present application, where the method includes:

receiving a data operation language (DML) request, wherein the DML request is used for requesting to operate on target data, and is irrelevant to a database type; generating a first transaction request for an OLTP database management system according to the DML request, wherein the first transaction request is used for requesting to operate on the target data in an OLTP database; sending the first transaction request to the OLTP database management system; acquiring a transaction log generated by the OLTP database management system in response to the first transaction request, wherein the transaction log comprises information of operations of the OLTP database management system on the target data; generating a second transaction request for the OLAP database management system according to the transaction log, wherein the second transaction request is used for requesting to operate on the target data in the OLAP database; and sending the second transaction request to the OLAP database management system.

In the method, since the DML request is independent of the database type, the DML request is used for requesting to operate on the target data, and because the OLTP database has transactional properties, a first transaction request for the OLTP database management system can be generated, and after the transaction log sent by the OLTP database management system is obtained, it can be determined that the first transaction request of the OLTP database must be successfully executed, so that a second transaction request can be generated according to the transaction log, and the second transaction request can be sent to the OLAP database management system, so that the OLAP database can be synchronized with the OLTP database, thereby realizing effective business fusion of the two databases and improving the use efficiency of the database.

In one possible design, the generating a first transaction request for an OLTP database management system from the DML request includes:

carrying out grammar analysis on the DML request to obtain a first grammar tree corresponding to the DML request, wherein the first grammar tree comprises at least one node, and any one node in the at least one node is used for storing the corresponding relation between a DML instruction keyword and a structured query language SQL instruction keyword; and generating the first transaction processing request according to the SQL keywords corresponding to the nodes in the first grammar tree.

By adopting the scheme, because the nodes in the grammar tree have the determined position relation, and any one of the at least one node is used for storing the corresponding relation between the DML instruction keywords and the SQL instruction keywords of the structured query language, the first transaction processing request can be accurately generated according to the SQL keywords corresponding to the nodes in the first grammar tree.

In one possible design, the second transaction request is a single hybrid transaction request, and the generating a second transaction request for the OLAP database management system from the transaction log includes:

a single hybrid transaction request for the OLAP database management system is generated from the transaction log, the hybrid transaction request requesting at least one type of data operation on the target data in the OLAP database.

With this scheme, only one hybrid transaction request is required to request at least one type of data operation on the target data, so that the OLAP database can be operated efficiently.

In one possible design, the second transaction request is at least one single type transaction request, and the generating a second transaction request for the OLAP database management system based on the transaction log includes:

Generating at least one single type transaction request for the OLAP database management system based on the transaction log, any single type transaction request being used to request one type of data operation on the target data in the OLAP database.

With this scheme, any single type transaction request is used to request one type of data operation to be performed on the target data, so that the OLAP database can be operated more accurately.

In a second aspect, an embodiment of the present application provides a data processing method, which may be applied to the data processing apparatus provided in the fourth aspect of the present application, where the method includes:

receiving a data query language, DQL, request for requesting query target data, the DQL request being independent of database type; acquiring corresponding transaction requests according to the DQL request, wherein the transaction requests comprise a first transaction request for an OLTP database management system or a second transaction request for an OLAP database management system, the first transaction request is used for requesting to query the target data stored in the OLTP database, and the second transaction request is used for requesting to query the target data stored in the OLAP database; and sending the transaction request to a database management system corresponding to the transaction request.

In the method, the DQL request is used for requesting the query target data and is irrelevant to the database type, so that the corresponding transaction request can be directly obtained according to the DQL request, and then the transaction request is sent to the corresponding database management system, so that the corresponding database management system can be flexibly selected, and the data processing efficiency is improved.

In one possible design, the obtaining the corresponding transaction request according to the DQL request includes:

acquiring an execution information set, wherein the execution information set comprises at least one piece of execution information, and any piece of execution information comprises a DQL request and a corresponding transaction processing request; and if the first execution information exists in the execution information set, the first execution information comprises the received DQL request, and a transaction processing request is acquired from the first execution information.

By adopting the scheme, as the execution information set is collected in advance, any execution information comprises the DQL request and the corresponding transaction request, when the DQL request is received, the execution information set can be directly searched for whether the first execution information exists, and then the transaction request corresponding to the DQL request is acquired.

In one possible design, if the first execution information does not exist in the execution information set, the method further includes:

generating the first transaction request of the OLTP database management system and the second transaction request of the OLAP database management system from the DQL request; sending the first transaction request to the OLTP database management system, determining a first performance index of an execution process of a corresponding transaction operation of the first transaction request, sending the second transaction request to the OLAP database management system, and determining a second performance index of an execution process of a corresponding transaction operation of the second transaction request; taking a transaction request corresponding to the highest performance index in the first performance index and the second performance index as a transaction request corresponding to the DQL request; and generating the execution information of the DQL request according to the DQL request and the transaction request corresponding to the DQL request.

With this arrangement, the first performance index and the second performance index are determined by actually executing the first transaction request and the second transaction request, so that the transaction request of the database management system corresponding to the higher performance index can be recorded in the execution information.

In one possible design, the first transaction request corresponds to a first process that creates a first cost-optimized CBO request for the OLTP database management system, the first CBO request to obtain a first CBO result, the first CBO result being the first performance index; the second transaction request corresponds to a second process, the second process creates a second CBO request of the OLAP database management system, the second CBO request is used to obtain a second CBO result, and the second CBO result is the second performance index; the first process is executed in parallel with the second process.

By adopting the scheme, the first process and the second process can be executed in parallel, so that the first performance index and the second performance index can be tested simultaneously, and the test performance can be evaluated more accurately by adopting the test scheme based on cost optimization, and the first performance index and the second performance index can be obtained more efficiently and accurately.

determining a query type of the DQL request; determining a database management system with matched query types of the DQL requests, wherein the matched database management system is the OLTP database management system or the OLAP database management system; and generating a transaction request corresponding to the matched database management system according to the DQL request, and sending the transaction request to the matched database management system.

By adopting the scheme, the database management system with the matched query types can be preset by introducing the query types, so that the database management system more suitable based on the query types can be generated according to the query characteristics of the first execution information even if the first execution information does not exist in the execution information set, thereby improving the query efficiency.

In one possible design, the generating the transaction request corresponding to the matched database management system according to the target DQL request includes:

carrying out grammar analysis on the target DQL request to obtain a second grammar tree corresponding to the DQL request, wherein the second grammar tree comprises at least one node, and any one node in the at least one node is used for storing the corresponding relation between the DQL instruction keywords and the structured query language SQL instruction keywords; and generating the second transaction processing request according to the SQL keywords corresponding to the nodes in the second grammar tree.

By adopting the scheme, because the nodes in the grammar tree have the determined position relation, and any one of the at least one node is used for storing the corresponding relation between the DQL instruction keywords and the SQL instruction keywords of the structured query language, the second transaction processing request can be accurately generated according to the SQL keywords corresponding to the nodes in the second grammar tree.

In a third aspect, an embodiment of the present application provides a data processing apparatus, including:

the acquisition module is used for receiving a data operation language (DML) request, wherein the DML request is used for requesting to operate on target data, and the DML request is irrelevant to the type of a database;

the processing module is used for generating a first transaction processing request for the OLTP database management system according to the DML request, wherein the first transaction processing request is used for requesting to operate on the target data in the OLTP database;

a sending module, configured to send the first transaction processing request to the OLTP database management system;

the obtaining module is further configured to obtain a transaction log generated by the OLTP database management system in response to the first transaction request, where the transaction log includes information that the OLTP database management system operates on the target data;

the processing module is used for generating a second transaction request for the OLAP database management system according to the transaction log, wherein the second transaction request is used for requesting to operate on the target data in the OLAP database;

and the sending module is used for sending the second transaction processing request to the OLAP database management system.

In one possible design, the processing module is specifically configured to:

In one possible design, the second transaction request is a single hybrid transaction request, and the processing module is specifically configured to:

In one possible design, the second transaction request is at least one single type transaction request, and the processing module is specifically configured to:

In a fourth aspect, embodiments of the present application provide a data processing apparatus, including:

an acquisition module for receiving a data query language, DQL, request for requesting query of target data, the DQL request being independent of database type;

and obtaining a corresponding transaction request according to the DQL request, the transaction request including a first transaction request for an OLAP database management system or a second transaction request for an OLAP database management system, the first transaction request being for requesting to query the target data stored in the OLAP database, the second transaction request being for requesting to query the target data stored in the OLAP database;

and the sending module is used for sending the transaction request to a database management system corresponding to the transaction request.

In one possible design, the acquisition module is specifically configured to:

In one possible design, the apparatus further includes a processing module, where if the first execution information does not exist in the execution information set, the processing module is specifically configured to:

In one possible design, the apparatus further includes a processing module, where if the first execution information does not exist in the execution information set, the processing module is further configured to:

determining a query type of the DQL request; determining a database management system with matched query types of the DQL requests, wherein the matched database management system is the OLTP database management system or the OLAP database management system; generating a transaction request corresponding to the matched database management system according to the DQL request;

the sending module is further configured to:

the transaction request is sent to the matching database management system.

In one possible design, the processing module is specifically configured to:

In a fifth aspect, there is provided an electronic device comprising: one or more processors; one or more memories; wherein the one or more memories store one or more computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any of the first and second aspects above.

In a sixth aspect, there is provided a computer readable storage medium comprising computer instructions which, when run on a computer, cause the computer to perform the method of any one of the first or second aspects above.

The advantages of the third aspect to the sixth aspect are described above with reference to the advantages of the first aspect or the second aspect, and the description is not repeated.

Drawings

FIG. 1 is a schematic diagram of a system architecture to which a data processing method according to an embodiment of the present application is applicable;

FIG. 2 is a schematic diagram of a server in a system architecture to which a data processing method according to an embodiment of the present application is applicable;

FIG. 3 is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

Fig. 4 is a flowchart illustrating specific steps of processing a DML request in a data processing method according to an embodiment of the present application;

FIG. 5 is a flowchart illustrating steps in a method for processing a DQL request according to an embodiment of the present application;

FIG. 6 is a flowchart illustrating steps in another embodiment of a data processing method according to the present disclosure;

fig. 7 is a flowchart illustrating specific steps for generating execution information in a data processing method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include, for example, "one or more" such forms of expression, unless the context clearly indicates to the contrary. It should also be understood that in embodiments of the present application, "one or more" refers to one or more than two (including two); "and/or", describes an association relationship of the association object, indicating that three relationships may exist; for example, a and/or B may represent: a alone, a and B together, and B alone, wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

The term "plurality" in the embodiments of the present application means two or more, and for this reason, "plurality" may be also understood as "at least two" in the embodiments of the present application. "at least one" may be understood as one or more, for example as one, two or more. For example, including at least one means including one, two or more, and not limiting what is included. For example, at least one of A, B and C is included, then A, B, C, A and B, A and C, B and C, or A and B and C may be included. Likewise, the understanding of the description of "at least one" and the like is similar. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/", unless otherwise specified, generally indicates that the associated object is an "or" relationship.

Unless stated to the contrary, the embodiments of the present application refer to ordinal terms such as "first," "second," etc., for distinguishing between multiple objects and not for defining a sequence, timing, priority, or importance of the multiple objects.

Databases in common use today fall into two categories: an online transaction (On-Line Transaction Processing, OLTP) database and an online analytical processing (On-Line Analytical Processing, OLAP) database. OLTP databases and OLAP databases are both of great interest in terms of business characteristics and technical application. In some scenarios, both databases are often deployed simultaneously. For example, in a Software-as-a-Service (SaaS) scenario, when the SaaS is applied, the manufacturer can uniformly deploy application Software, and a user can subscribe the SaaS to the SaaS manufacturer according to actual requirements, so as to obtain related business services of the SaaS. Thus, the user does not need to locally deploy the infrastructure of the database, so that the operation and maintenance cost can be effectively saved.

SaaS is developed to be mature at present, multi-lease sharing can be realized, and under the condition of multi-lease sharing, in order to meet the business requirements of different users, an OLTP database and an OLAP database can be deployed on the SaaS at the same time. The database of the SaaS stores data records of a plurality of users in the form of Physical tables (Physical tables), and the data records are presented to the users in the form of Logical tables (Logical tables). When the data processing method provided by the application is adopted, the data processing request of the logic table can be sent to the SaaS server through the user side of the user, and the SaaS server generates the corresponding transaction processing request of the physical table according to the data processing request. The physical table refers to a data table actually created in a database of the SaaS and used for data record storage, the logical table refers to a data table customized by a user according to own service, the logical table does not exist in the database of the SaaS, and the conversion from a data processing request to a transaction processing request can be realized through mapping between the logical table and the physical table. Wherein, the data processing request can be a data operation language (Data Manipulation Language, DML) request or a data query language (Data Query Language, DQL) request, and the DML request or the DQL request is irrelevant to the database type, and can access the physical table through generating corresponding transaction requests of different types of database management systems, thereby realizing the processing of the DML request or the DQL request.

Specifically, the physical table may be as shown in Table 1-1 below:

table 1-1: physical table (table name Data)

Wherein GUID, tenant, object, field, field2, field3 and Field4 are fields of a Data table Data, specifically, GUID is a number; tenant is a user name, e.g., tenant_a is a specific user name; objects are table objects, such as Customer, company refers to the table name of a particular logical table of table objects, and fields 1, 2, 3, and 4 are non-type fields, which may store various types of fields. The user can flexibly customize the fields to map to fields 1-4 in the physical table.

It should be noted that, an Index table (table name index_data) of the physical table may also be established according to the type of the Index in each logical table in the table object, as in tables 1-2:

TABLE 1-2 Index Table (Table name Index_Data)

Wherein GUID, tenant, object, string, number is a field of the Index table index_data. String is the index of the logical table Company and Number is the index of the logical table Customer.

The database of the SaaS may further describe the basic information of the table object through the object metadata table, and describe the basic information of each field in the table object through the field metadata table, where the object metadata table may be shown in table 2-1, and the field metadata table may be shown in table 2-2.

TABLE 2-1 object metadata Table (Table name Objects)

Wherein Name, tenant is a field of the object metadata table Objects, name represents a table Name of a table object, tenant represents a user Name.

Table 2-2 field metadata Table (Table name Fields)

Wherein Name, tenant, object, type, field _No and Index are Fields of the field metadata table Fields. Name indicates the Field Name of the Field defined in the table Object, tenant indicates the user Name corresponding to the Field, object indicates the table Object Name to which the Field belongs, type indicates the specific Type of the Field, field_no indicates the No-Type Field number to which the Field is mapped in the physical table Data, e.g., field_no 1 corresponds to Field1, and Index indicates whether the Field is an Index. Obviously, the user can modify the definitions in the field metadata table as desired without affecting other users.

Logical tables can be presented to users by tables 1-1 to 1-2, and tables 2-1 to 2-2, i.e., table 3-1 and table 3-2, with table 3-1 being a user logical table (table name Customer) and table 3-2 being an organization logical table (table name Company).

TABLE 3-1 user logic List (Table name Customer)

Wherein name, id, type and company are fields defined in the table Customer, which have been defined in table 2-2, name represents a user name, id represents a user number, type represents a user type, and company represents a organization name corresponding to the user.

TABLE 3-2 organization logic Table (Table name company)

Wherein name, code, category and staff_num are fields defined in table company, and have been defined in table 2-2, name represents an organization name, code represents an organization code, category represents an organization type, and staff_num represents the number of persons of an organization.

It should be noted that, tables 1-1 to 1-2 and tables 2-1 to 2-2 are data tables actually defined in the database, and tables 3-1 and 3-2 are not data tables actually defined in the database, and may be presented to the user, so that the user can operate conveniently. In the actual application process, the virtual access of the user to the table 3-1 to 3-2 can be converted into the actual access to the table 1-1 to the table 1-2, but the process is not visible to the user.

According to the example of the SaaS, for the multi-lease sharing scene of the SaaS, the business requirements of different users can be met and different data processing requests can be processed by deploying the OLTP database and the OLTP database at the same time.

The following describes in detail a data processing method provided in the present application, and the method may be applied to a system architecture shown in fig. 1, where the system architecture shown in fig. 1 includes a user side 100, a server side 200, and a storage side 300. It should be understood that the system architecture shown in fig. 1 is only one example, and in some embodiments may have more or fewer components than in the system architecture shown in fig. 1, may combine two or more components, or may have a different configuration of components. The various components shown in fig. 1 may be implemented in hardware including one or more hardware, software, or a combination of hardware and software. The system architecture to which the data processing method is applicable may also include only the user terminal 100 and the server terminal 200, and the function of the storage terminal 300 may be transferred to the server terminal 200. The system architecture shown in fig. 1 is specifically as follows:

The client 100 is configured to obtain a data processing request, for example, the data processing request may be generated according to a database operation command input by a user, or may be received from another device. The data processing request may be a DML request or a DQL request. The client 100 may also be configured to send a data processing request to the server 200. In the present application, specific physical devices for implementing the client 100 are not limited, and physical devices capable of initiating a data processing request may be all physical devices, for example, the client 100 may be a host device.

The server 200 is configured to obtain a data processing request from the client 100, and process the data processing request. It should be noted that, in the process of processing the data processing request, the database may be accessed. The system architecture shown in fig. 1 may deploy both OLTP and OLAP databases. The server 200 may process the data processing request according to the OLTP database and/or OLAP database. The specific physical device for implementing the server 200 is not limited in this application, for example, the server 200 may be a server.

The storage 300 is used to deploy OLTP databases and OLAP databases, run a database management system (Database Management System, DBMS), and store log files for the databases. It should be noted that in the example of fig. 1, only one storage 300 is shown, and this storage 300 deploys both the OLTP database and the OLAP database. There may be other flexible deployments, such as two storage 300, where one storage 300 deploys an OLTP database and the other deploys an OLAP database. In some other examples, the database may be deployed on a server without the storage 300. The specific physical device implementing the server 200 is not limited in this application, for example, the storage 300 may be a host device running random access memory (Random Access Memory, RAM).

In the system architecture shown in fig. 1, the specific architecture of the server 200 may also be as shown in fig. 2. The functional modules in the server 200 may specifically be as follows:

the execution information generator 201 may include a processing request parser 2011, a processing request translator 2012, and an execution information cache 2013. The processing request parser 2011 is configured to parse a data processing request, for example, parse the data processing request into a syntax tree, and may also have other parsing manners and parsing contents, for example, parse the data processing request into a regular expression, etc., where the data processing request may be a processing request for operating a user logic table; the processing request translator 2012 is configured to translate the parsing content (for example, the syntax tree in fig. 2) to obtain a corresponding transaction request, where the transaction request may be a processing request for operating a physical table of a user; the execution information buffer 2013 is configured to store execution information of the corresponding transaction process of the transaction request, and the execution information may have various formats, such as a data processing request and a transaction request of a corresponding database. In some cases, the execution information generator 201 may also include a semantic analyzer 2014 for determining the query type of the DQL request.

A transaction executor 202, configured to connect the databases and send a transaction request to the corresponding database; the transaction executor 202 may include a DML executor 2021 and a DQL executor 2022, the DML executor to send the DML type transaction request and the DQL executor to send the DQL type transaction request.

The data synchronizer 203 may include a log parser 2031 and an OLAP executor 2032. The log parser is configured to obtain a transaction log of the OLTP database management system, and generate a transaction request of the OLAP database management system according to the transaction log of the OLTP database management system, so that the OLTP database and the OLAP database are kept consistent.

The execution optimizer 204 is configured to select a transaction request from the transaction requests of the OLAP database management system and the transaction requests of the OLTP database management system, and may determine performance indexes of the transaction processes corresponding to the two transaction requests respectively according to a Cost-optimized (Cost-Based Optimization, CBO) manner. The execution optimizers 204 may include an OLAP optimizer 2041 and an OLTP optimizer 2042 for sending respective transaction requests and CBO requests, respectively, to determine performance metrics for the transaction processes corresponding to the transaction requests for the two databases.

The server 200 shown in fig. 2 may be used to perform a data processing method provided in the present application. Referring to fig. 3, a data processing method provided in the present application is described in detail below.

Step 301: a data processing request is received.

If the data processing request is a DML request, executing a processing method of the DML request, namely, steps 302 to 306; if the data processing request is a DQL request, the processing method of the DQL request is performed, i.e., steps 307-308. The DML request is used for requesting to operate on target data, and the DML request is irrelevant to the type of the database; the DQL request is used to request query target data, the DQL request being independent of database type.

Step 302: a first transaction request for an OLTP database management system is generated based on the DML request.

Step 303: a first transaction request is sent to an OLTP database management system.

Step 304: a transaction log generated by an OLTP database management system in response to a first transaction request is obtained.

The transaction log comprises information of operations of the OLTP database management system on target data.

Step 305: a second transaction request for the OLAP database management system is generated based on the transaction log.

Step 306: a second transaction request is sent to the OLAP database management system.

Step 307: based on the DQL request, a corresponding transaction request is obtained.

The transaction request includes a first transaction request for an OLTP database management system or a second transaction request for an OLAP database management system.

Step 308: and sending the transaction request to a database management system corresponding to the transaction request.

It should be noted that, in steps 302 to 306, the first transaction request is used for requesting to operate on the target data in the OLTP database, and the second transaction request is used for requesting to operate on the target data in the OLAP database; whereas in steps 307-308, a first transaction request is used to request a query of target data stored in the OLTP database and a second transaction request is used to request a query of target data stored in the OLAP database.

Steps 302 to 306 are processing methods of DML requests, and steps 307 to 308 are processing methods of DQL requests, and related steps are described in detail below.

The processing method of the DML request comprises the following steps:

in step 302, one possible implementation may be as follows:

and carrying out grammar analysis on the DML request to obtain a first grammar tree corresponding to the DML request, and generating a first transaction processing request according to a structured query language (Structured Query Language, SQL) keyword corresponding to the node in the first grammar tree.

The first syntax tree includes at least one node, and any one of the at least one node is configured to store a correspondence between DML instruction keywords and structured query language SQL instruction keywords.

For example, for the examples of tables 1-1 through 3-2, the DML request is specifically:

INSERT INTO Customer(name,id,type,company)VALUES(‘Mary’，10022，‘VIP1’,‘CompanyA’)；

since the DML request is a processing request for the user logic table, and the user logic table is not actually created in the database, the DML request needs to be converted into a first transaction request to operate on the physical table Data, where the first transaction request is specifically:

INSERT INTO Data(GUID,Tenant,Objects,Filed1,Filed2,Filed3,Filed4)VALUES(GUID,‘Tenant_A’,Customer,‘Mary’，10022，‘VIP1’,‘CompanyA’)；

the DML request and the first transaction request are both composed of a plurality of keywords, the node of the first syntax tree stores a correspondence between the DML instruction keywords and the SQL instruction keywords, and the tree structure of the first syntax tree may also indicate the order of the SQL instruction keywords, so that the DML request can be converted into the first transaction request through the first syntax tree, for example, the node in the first syntax tree stores a correspondence between the file 1 and the name, and the file 1 can be converted into the name. The form of the first syntax tree is not limited in the present application, and the first transaction processing request may be generated, and the form from the DML request to the first transaction processing request may be implemented.

In step 304, in one possible implementation manner, the transaction log of the OLTP database management system may be monitored, and a change log in the transaction log of the OLTP database management system after the execution of step 303 is obtained, where the change log is used as the transaction log generated by the OLTP database management system in response to the first transaction request; the OLTP database management system may also receive a corresponding transaction log sent by the OLTP database management system after the first transaction request is responded, which is not limited herein.

Since the OLTP database management system satisfies the transactional, i.e., atomic, consistency, isolation, and persistence (ACID). Thus, if step 304 is performed successfully, it is stated that the OLTP database management system must have performed successfully, i.e., the OLTP database has performed the data operation as expected by the first transaction request. Even if 304 fails to execute, the OLTP database can be rolled back to the state of step 303 to execute again, and no data pollution is caused. Therefore, the first transaction request is sent first, and then the second transaction request is sent, so that the fault tolerance of the data operation can be improved.

In one possible implementation of step 305, the second transaction request is a single hybrid transaction request, and step 305 may be specifically performed as follows:

from the transaction log, a single hybrid transaction request is generated for the OLAP database management system.

The hybrid transaction request is for requesting at least one type of data operation on target data in an OLAP database, for example, the at least one type of data operation may be: at least one of a data insertion operation, a data deletion operation, and a data modification operation.

In another possible implementation of step 305, the second transaction request is at least one single type transaction request, and step 305 may be specifically performed in the following manner:

at least one single type transaction request for the OLAP database management system is generated from the transaction log.

Wherein, any single type transaction request is used for requesting to perform one type of data operation on the target data in the OLAP database, and the data operation of any single type transaction request can be: any one of a data insertion operation, a data deletion operation, and a data modification operation.

If the system architecture shown in fig. 1 is adopted, the service end 200 adopts the architecture shown in fig. 2, and a specific interaction flow for processing the DML request may be as shown in fig. 4.

Step 401: the client 100 sends a DML request to the processing request parser 2011 of the server 200.

Step 402: the processing request parser 2011 of the server 200 parses the DML request to obtain a first syntax tree corresponding to the DML request.

Step 403: the processing request parser 2011 of the server 200 sends the first syntax tree to the processing request translator 2012 of the server 200.

Step 404: the processing request translator 2012 of the server 200 generates a first transaction request according to the SQL key corresponding to the node in the first syntax tree.

Step 405: the processing request translator 2012 of the server 200 sends the first transaction request to the transaction executor 202 of the server 200.

Step 406: transaction executor 202 of server 200 sends a first transaction request to the OLTP database management system in storage 300.

Step 407: the OLTP database management system performs a first transaction processing request.

Step 408: the OLTP database management system sends the execution result of the first transaction request to the transaction executor 202 of the server 200.

Step 409: the transaction executor 202 of the server 200 determines whether the first transaction request is executed successfully according to the execution result of the first transaction request.

If the execution is unsuccessful, execute step 410; otherwise, step 411 is performed.

Step 410: the transaction executor 202 of the server 200 returns an execution failure message to the user 100.

Step 411: the transaction executor 202 of the server 200 returns the execution result of the first transaction request to the user 100.

Step 412: and the OLTP database management system generates a transaction log corresponding to the first transaction request according to the execution result of the first transaction request.

Step 413: the OLTP database management system sends a transaction log corresponding to the first transaction request to the data synchronizer 203 of the server 200.

Step 414: the data synchronizer 203 of the server 200 generates a second transaction request for the OLAP database management system based on the transaction log.

Step 415: the data synchronizer 203 of the server 200 sends a second transaction request to the transaction executor 202 of the server 200.

Step 416: the transaction executor 202 of the server 200 sends a second transaction request to the OLAP database management system.

Step 417: the OLAP database management system performs a second transaction request.

The processing method of the DQL request comprises the following steps:

one possible implementation of step 307 may be as follows:

and acquiring an execution information set, and acquiring a transaction processing request from the first execution information if the first execution information exists in the execution information set.

The execution information set comprises at least one execution information, any one of the execution information comprises a DQL request and a corresponding transaction request, and the first execution information comprises the received DQL request.

If the first execution information exists in the execution information set, when the system architecture shown in fig. 1 is adopted and the service end 200 adopts the architecture shown in fig. 2, the DQL request can be processed according to the interaction flow shown in fig. 5:

step 501: the user 100 sends a DQL request to the processing request parser 2011 of the server 200.

Step 502: the processing request parser 2011 of the server 200 determines that the first execution information exists in the execution information set.

Step 503: the processing request parser 2011 of the server 200 sends the transaction request in the first execution information to the transaction executor 202 of the server 200.

If the transaction request corresponding to the DQL request in the first execution information is the first transaction request, executing steps 504 to 506; otherwise, if the transaction request corresponding to the DQL request in the first execution information is the second transaction request, steps 507 to 509 are executed.

In a method of processing a DQL request, a first transaction request is used to request a query of target data stored in an OLTP database and a second transaction request is used to request a query of target data stored in an OLAP database.

Step 504: the transaction executor 202 of the server 200 sends a first transaction request to the OLTP database management system.

Step 505: the OLTP database management system performs a first transaction processing request.

Step 506: the OLTP database management system sends the execution result of the first transaction request to the transaction executor 202 of the server 200.

Step 507: the transaction executor 202 of the server 200 sends a second transaction request to the OLAP database management system.

Step 508: the OLAP database management system performs a second transaction request.

Step 509: the OLAP database management system sends the execution result of the second transaction request to the transaction executor 202 of the server 200.

If the first execution information does not exist in the execution information set, the DQL request can be further processed as follows:

determining a query type of the DQL request; a database management system that determines that the query types of the DQL requests match; and generating a transaction request corresponding to the matched database management system according to the DQL request, and sending the transaction request to the matched database management system.

For example, the query types of the DQL request may include joint query, conditional query, and grouping query, and the database management system corresponding to each query type may be preset according to the characteristics of the database management system, such as grouping query corresponding to OLAP database management system, because the grouping query may count columns of a large number of data records in the data table. And the query result sets are grouped, and the OLAP database management system contains more indexes and adopts a parallel technology, so that the OLAP database management system is adopted to execute corresponding transaction processing requests, and the performance of the transaction processing executing process can be higher.

In a possible implementation manner, the step of generating the transaction request corresponding to the matched database management system may specifically be as follows:

The DQL request is subjected to grammar parsing to obtain a second grammar tree corresponding to the DQL request, wherein the second grammar tree comprises at least one node, and any one node in the at least one node is used for storing the corresponding relation between the DQL instruction keywords and the structured query language SQL instruction keywords; and generating the second transaction processing request according to the SQL keywords corresponding to the nodes in the second grammar tree.

The process of generating the second transaction request is substantially the same as the process of generating the first transaction request, and reference may be made to the description of generating the first transaction request, which is not repeated herein.

For the case that the first execution information does not exist in the execution information set, there may be various embodiments of processing the DQL request, for example, it may be preset that the database management system corresponding to the transaction request is an OLAP database management system, and one database management system is selected randomly.

For the situation that the first execution information does not exist in the execution information set, if the system architecture shown in fig. 1 is adopted and the server 200 shown in fig. 2 is adopted, the server 200 shown in fig. 2 may further be provided with a semantic analyzer 2014, and the DQL request may be processed according to the interaction flow shown in fig. 6:

Step 601: the user 100 sends a DQL request to the processing request parser 2011 of the server 200.

Step 602: the processing request parser 2011 of the server 200 determines that the first execution information does not exist in the execution information set.

Step 603: the processing request parser 2011 of the server 200 sends the DQL request to the semantic analyzer 2014 of the server 200.

Step 604: the semantic analyzer 2014 of the server 200 determines the query type of the DQL request and a database management system that determines that the query type of the DQL request matches.

If the matched database management system is an OLTP database management system, executing steps 605 to 609; if the matching database management system is an OLAP database management system, steps 610 to 614 are performed.

Step 605: the processing request translator 2012 of the server 200 generates a first transaction request corresponding to the OLTP database management system according to the DQL request.

Step 606: the processing request translator 2012 of the server 200 sends the first transaction request to the transaction executor 202 of the server 200.

Step 607: the transaction executor 202 of the server 200 sends a first transaction request to the OLTP database management system.

Step 608: the OLTP database management system performs a first transaction processing request.

Step 609: the OLTP database management system sends the execution result of the first transaction request to the transaction executor 202 of the server 200.

Step 610: the processing request translator 2012 of the server 200 generates a second transaction request corresponding to the OLAP database management system according to the DQL request.

Step 611: the processing request translator 2012 of the server 200 sends the second transaction request to the transaction executor 202 of the server 200.

Step 612: the transaction executor 202 of the server 200 sends a second transaction request to the OLAP database management system.

Step 613: the OLAP database management system performs a second transaction request.

Step 614: the OLAP database management system sends the execution result of the second transaction request to the transaction executor 202 of the server 200.

It should be noted that, for the case where the first execution information does not exist in the execution information set, the execution information of the DQL request may also be generated according to the following steps:

step (1): the first transaction request of the OLTP database management system and the second transaction request of the OLAP database management system are generated from the DQL request.

Step (2): and sending the first transaction request to the OLTP database management system, determining a first performance index of the execution process of the corresponding transaction operation of the first transaction request, sending the second transaction request to the OLAP database management system, and determining a second performance index of the execution process of the corresponding transaction operation of the second transaction request.

Step (3): and taking the transaction request corresponding to the highest performance index in the first performance index and the second performance index as the transaction request corresponding to the DQL request.

Step (4): and generating the execution information of the DQL request according to the DQL request and the transaction request corresponding to the DQL request.

The performance index in the steps (1) to (4) is not limited herein. For example, the execution duration, memory footprint, etc. of the transaction operation may be performed.

The first performance index and the second performance index may be obtained as follows:

creating a first process and a second process; and executing the first process and the second process.

Wherein the first transaction request corresponds to a first process that creates a first Cost-based optimization (Cost-Based Optimization, CBO) request of the OLTP database management system, the first CBO request being for obtaining a first CBO result, the first CBO result being the first performance index; the second transaction request corresponds to a second process, the second process creates a second CBO request of the OLAP database management system, the second CBO request is used to obtain a second CBO result, and the second CBO result is the second performance index.

It should be noted that, the execution of the first process and the second process is not in a constrained relationship, and may be executed in parallel or asynchronously. When the first process and the second process are executed in parallel, the testing duration of the two performance indexes is overlapped, and the total duration of performance index testing is reduced.

For the situation that the first execution information does not exist in the execution information set, if the system architecture shown in fig. 1 is adopted, and the service end 200 adopts the architecture shown in fig. 2, the execution information of the DQL request can be obtained according to the interaction flow shown in fig. 7:

step 701: the user 100 sends a DQL request to the processing request parser 2011 of the server 200.

Step 702: the processing request parser 2011 of the server 200 sends the DQL request to the processing request translator 2012 of the server 200.

Step 703: the processing request translator 2012 of the server 200 generates a first transaction request corresponding to the OLTP database management system and a second transaction request corresponding to the OLAP database management system according to the DQL request.

Step 704: the processing request translator 2012 of the server 200 sends the first transaction request and the second transaction request to the execution optimizer 204 of the server 200.

Step 705: the execution optimizer 204 of the server 200 creates a first CBO request for a first transaction request and a second CBO request for a second transaction request.

Step 706: the execution optimizer 204 of the server 200 sends a first transaction request and a first CBO request to the OLTP database management system.

Step 707: the OLTP database management system executes the first transaction request and the first CBO request to obtain a first CBO result.

Step 708: the OLTP database management system sends the first CBO result to the execution optimizer 204 of the server 200.

Step 709: the execution optimizer 204 of the server 200 sends a second transaction request and a second CBO request to the OLAP database management system.

Step 710: the OLAP database management system executes the second transaction request and the second CBO request to obtain a second CBO result.

Step 711: the OLTP database management system sends the second CBO result to the execution optimizer 204 of the server side 200.

The order of execution between steps 706 to 708 and 709 to 711 is not limited, and steps 706 to 708 may be executed in parallel with steps 709 to 711.

Step 712: the execution optimizer 204 of the server 200 takes the transaction request corresponding to the highest performance CBO result in the first CBO result and the second CBO result as the transaction request corresponding to the DQL request, and generates the execution information of the DQL request.

Step 713: the execution optimizer 204 of the server 200 sends the execution information of the DQL request to the execution information cache 2013 of the server 200.

As shown in fig. 8, an embodiment of the present application provides a data processing apparatus. When the apparatus shown in fig. 8 is used to process DML requests, the modules of the apparatus are specifically configured to:

an obtaining module 801, configured to receive a DML request in a data operation language, where the DML request is used to request to operate on target data, and the DML request is independent of a database type;

a processing module 802, configured to generate, according to the DML request, a first transaction request for an OLTP database management system, where the first transaction request is used to request to operate on the target data in an OLTP database;

a sending module 803 configured to send the first transaction processing request to the OLTP database management system;

the obtaining module 801 is further configured to obtain a transaction log generated by the OLTP database management system in response to the first transaction request, where the transaction log includes information that the OLTP database management system operates on the target data;

the processing module 802 is configured to generate, according to the transaction log, a second transaction request for the OLAP database management system, where the second transaction request is used to request to operate on the target data in the OLAP database;

The sending module 803 is configured to send the second transaction request to the OLAP database management system.

In one possible design, the processing module 802 is specifically configured to:

In one possible design, the second transaction request is a single hybrid transaction request, and the processing module 802 is specifically configured to:

In one possible design, the second transaction request is at least one single type transaction request, and the processing module 802 is specifically configured to:

When the apparatus shown in fig. 8 is used to process DQL requests, the modules of the apparatus are specifically configured to:

an acquisition module 801 for receiving a data query language DQL request for requesting query target data, the DQL request being independent of database type;

and the sending module 803 is configured to send the transaction request to a database management system corresponding to the transaction request.

In one possible design, the obtaining module 801 is specifically configured to:

In one possible design, the apparatus further includes a processing module 802, where if the first execution information does not exist in the execution information set, the processing module 802 is specifically configured to:

In one possible design, the apparatus further includes a processing module 802, where if the first execution information does not exist in the execution information set, the processing module 802 is further configured to:

The sending module 803 is further configured to:

the transaction request is sent to the matching database management system.

The embodiment of the application also provides an electronic device, which may have a structure as shown in fig. 9, and may be a computer device or a chip system capable of supporting the computer device to implement the method.

The electronic device as shown in fig. 9 may comprise at least one processor 901, said at least one processor 901 being adapted to be coupled with a memory, read and execute instructions in said memory to implement the steps of data processing provided by embodiments of the present application. Optionally, the electronic device may further include a communication interface 902 for supporting the electronic device for signaling or data reception or transmission. A communication interface 902 in the electronic device may be used to enable interaction with other electronic devices. The processor 901 may be used to implement steps of performing data processing by an electronic device. Optionally, the electronic device may further comprise a memory 903, in which computer instructions are stored, the memory 903 may be coupled to the processor 901 and/or the communication interface 902 for supporting the steps of the processor 901 invoking the computer instructions in the memory 903 to perform data processing; in addition, the memory 903 may be used to store data related to the method embodiments of the present application, for example, to store data, instructions necessary to support the interaction performed by the communication interface 902, and/or to store configuration information necessary for the electronic device to perform the methods described in the embodiments of the present application.

Embodiments of the present application also provide a computer readable storage medium, where computer instructions are stored, where the computer instructions, when executed by a computer, may cause the computer to perform the method involved in any one of the possible designs of the method embodiments and the method embodiments described above. In the embodiment of the present application, the computer readable storage medium is not limited, and may be, for example, RAM (random-access memory), ROM (read-only memory), or the like.

The present application also provides a chip that may include a processor and interface circuitry for performing the methods referred to in any one of the possible implementations of the method embodiments described above, wherein "coupled" means that the two components are directly or indirectly joined to each other, which may be fixed or movable.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, optical fiber), or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

The steps of a method or algorithm described in the embodiments of the present application may be embodied directly in hardware, in a software element executed by a processor, or in a combination of the two. The software elements may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In an example, a storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC, which may reside in a terminal device. In the alternative, the processor and the storage medium may reside in different components in a terminal device.

These computer instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

Although the invention has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof can be made without departing from the scope of the invention. Accordingly, the specification and drawings are merely exemplary illustrations of the present invention as defined in the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the invention. It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method of data processing, comprising:

receiving a data operation language (DML) request, wherein the DML request is used for requesting to operate on target data, and is irrelevant to a database type;

generating a first transaction request for an OLTP database management system according to the DML request, wherein the first transaction request is used for requesting to operate on the target data in an OLTP database;

Sending the first transaction request to the OLTP database management system;

acquiring a transaction log generated by the OLTP database management system in response to the first transaction request, wherein the transaction log comprises information of operations of the OLTP database management system on the target data;

generating a second transaction request for the OLAP database management system according to the transaction log, wherein the second transaction request is used for requesting to operate on the target data in the OLAP database;

and sending the second transaction request to the OLAP database management system.

2. The method of claim 1, wherein generating a first transaction request for an OLTP database management system based on the DML request comprises:

carrying out grammar analysis on the DML request to obtain a first grammar tree corresponding to the DML request, wherein the first grammar tree comprises at least one node, and any one node in the at least one node is used for storing the corresponding relation between a DML instruction keyword and a structured query language SQL instruction keyword;

and generating the first transaction processing request according to the SQL keywords corresponding to the nodes in the first grammar tree.

3. The method of claim 1 or 2, wherein the second transaction request is a single hybrid transaction request, the generating a second transaction request for an OLAP database management system from the transaction log comprising:

4. The method of claim 1 or 2, wherein the second transaction request is at least one single type of transaction request, the generating a second transaction request for an OLAP database management system from the transaction log comprising:

5. A method of data processing, comprising:

Receiving a data query language, DQL, request for requesting query target data, the DQL request being independent of database type;

acquiring corresponding transaction requests according to the DQL request, wherein the transaction requests comprise a first transaction request for an OLTP database management system or a second transaction request for an OLAP database management system, the first transaction request is used for requesting to query the target data stored in the OLTP database, and the second transaction request is used for requesting to query the target data stored in the OLAP database;

and sending the transaction request to a database management system corresponding to the transaction request.

6. The method of claim 5, wherein the obtaining the corresponding transaction request from the DQL request comprises:

acquiring an execution information set, wherein the execution information set comprises at least one piece of execution information, and any piece of execution information comprises a DQL request and a corresponding transaction processing request;

and if the first execution information exists in the execution information set, the first execution information comprises the received DQL request, and a transaction processing request is acquired from the first execution information.

7. The method of claim 6, wherein if the first execution information is not present in the execution information set, further comprising:

generating the first transaction request of the OLTP database management system and the second transaction request of the OLAP database management system from the DQL request;

sending the first transaction request to the OLTP database management system, determining a first performance index of an execution process of a corresponding transaction operation of the first transaction request, sending the second transaction request to the OLAP database management system, and determining a second performance index of an execution process of a corresponding transaction operation of the second transaction request;

taking a transaction request corresponding to the highest performance index in the first performance index and the second performance index as a transaction request corresponding to the DQL request;

and generating the execution information of the DQL request according to the DQL request and the transaction request corresponding to the DQL request.

8. The method of claim 7, wherein,

the first transaction processing request corresponds to a first process, the first process creates a first cost-optimized CBO request of the OLTP database management system, the first CBO request is used for obtaining a first CBO result, and the first CBO result is the first performance index;

The second transaction request corresponds to a second process, the second process creates a second CBO request of the OLAP database management system, the second CBO request is used to obtain a second CBO result, and the second CBO result is the second performance index;

the first process is executed in parallel with the second process.

9. The method of any of claims 6 to 8, further comprising, if the first execution information is not present in the set of execution information:

determining a query type of the DQL request;

determining a database management system with matched query types of the DQL requests, wherein the matched database management system is the OLTP database management system or the OLAP database management system;

and generating a transaction request corresponding to the matched database management system according to the DQL request, and sending the transaction request to the matched database management system.

10. The method of claim 9, wherein generating the transaction request corresponding to the matching database management system from the target DQL request comprises:

the DQL request is subjected to grammar parsing to obtain a second grammar tree corresponding to the DQL request, wherein the second grammar tree comprises at least one node, and any one node in the at least one node is used for storing the corresponding relation between the DQL instruction keywords and the structured query language SQL instruction keywords;

And generating the second transaction processing request according to the SQL keywords corresponding to the nodes in the second grammar tree.

11. A data processing apparatus, comprising:

12. A data processing apparatus, comprising:

13. An electronic device, the electronic device comprising: one or more processors; one or more memories; wherein the one or more memories store one or more computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-4 or 5-10.

14. A computer readable storage medium comprising computer instructions which, when run on a computer, cause the computer to perform the method of any of claims 1-4 or 5-10.