CN107451220B

CN107451220B - Distributed NewSQL database system

Info

Publication number: CN107451220B
Application number: CN201710581195.7A
Authority: CN
Inventors: 晋彤
Original assignee: Yunrun Da Data Service Co ltd
Current assignee: Yunrun Da Data Service Co ltd
Priority date: 2016-09-21
Filing date: 2017-07-17
Publication date: 2020-06-09
Anticipated expiration: 2037-07-17
Also published as: CN107391653B; CN107402990A; CN107402995B; CN107402991A; CN107451219A; CN107491485B; CN107480198A; CN107463637A; CN107491345A; CN107402987B; CN107491485A; CN107329837A; CN107368575A; CN107368575B; CN107402989A; CN107451214B; CN107402992A; CN107480198B; CN107463637B; CN107247808B

Abstract

The invention discloses a distributed NewSQL database system, which comprises: the JDBC/ODBC interface unit is used for receiving a user request sent to the Master unit by a user and returning a result returned by the Master unit to the user; the Master unit is used for accessing a user request through the JDBC/ODBC interface unit, coordinating data communication among a plurality of processors and managing the whole flow, and preferentially sending the user request to the SQLPLaner unit; the SQLPLaner unit is used for analyzing the user request, compiling and customizing an execution plan according to the user request; and the Worker unit is used for executing the plan in parallel to obtain data, merging and summarizing the data and returning the data to the Master unit. The invention supports standard JDBC interface, simplifies ETL process and reduces cost.

Description

Distributed NewSQL database system

Technical Field

The invention relates to the technical field of big data, in particular to a distributed NewSQL database system.

Background

HBase is currently one of the best known distributed NoSQL databases in the Hadoop ecosystem. The design concept is derived from the Bigtable of Google. HBase provides only the access interface of the Java Api, which encapsulates a large number of data manipulation interfaces. But its mode of operation and method is the Hbase specific Api mode. Completely different from the JDBC/ODBC mode which people are used to. As more and more applications attempt to migrate to hbases, defects in hbases become more and more exposed: the user needs to access the HBase through API programming, the use cost of complex applications is too high, and the HBase cannot be used by many complex applications.

Disclosure of Invention

The embodiment of the invention aims to provide a distributed NewSQL database, which supports a standard JDBC interface, simplifies an ETL process and reduces the cost.

In order to achieve the above object, an embodiment of the present invention provides a distributed NewSQL database system, including:

the JDBC/ODBC interface unit is used for receiving a user request sent to the Master unit by a user and returning a result returned by the Master unit to the user;

the Master unit is used for accessing a user request through the JDBC/ODBC interface unit, coordinating data communication among a plurality of processors and managing the whole flow, and preferentially sending the user request to the SQLPLaner unit;

the SQLPLaner unit is used for analyzing the user request, compiling and customizing an execution plan according to the user request;

and the Worker unit is used for executing the plan in parallel to obtain data, merging and summarizing the data and returning the data to the Master unit.

Further, the method also comprises the following steps: and the distributed transaction management unit is used for coordinating multiple parties designed in the plan to finish distributed transaction management.

Further, the method also comprises the following steps: the hbase unit and the SOLR unit are used for storing data;

the Worker unit is further used for acquiring data through the hbase unit and the SOLR unit.

Further, the method also comprises the following steps: a monitor for taking charge of metadata management and for monitoring the load of regions of the hbase unit, avoiding that a specific Region is overloaded too high, and reallocating regions by using a Coprocessor module of the hbase unit;

the monitor is connected with the Master unit.

Further, the analyzing the user request, compiling and customizing the execution plan according to the user request specifically includes:

judging whether a pre-stored SQL statement corresponding to the SQL statement exists in the shared cache pool, if so, outputting an execution plan corresponding to the SQL statement, and if not, outputting an execution plan corresponding to the SQL statement

Syntax checking is carried out on the SQL statement, if syntax errors return error information to a user, otherwise,

semantic check is carried out on the SQL statement, if the semantic is wrong, error information is returned to the user, otherwise,

carrying out view and expression conversion on the SQL statement to obtain a corresponding conversion result;

selecting an optimizer according to the conversion result to obtain a corresponding optimizer selection result;

selecting a corresponding data connection mode and a corresponding connection sequence according to the selection result of the optimizer;

selecting a searched path according to the connection mode and the connection sequence;

and generating an execution plan according to the search path, and outputting the execution plan.

Further, the JDBC/ODBC interface unit includes:

the JDBC application program module is used for carrying out interactive operation with a user and calling a JDBC object method to give an SQL statement and extract a result;

the JDBC driver manager module is used for loading and calling the JDBC driver module for the JDBC application program module;

and the JDBC driver module is used for executing the calling of the JDBC object method, sending an SQL request to the Master unit and returning the result returned by the Master unit to the JDBC application program module.

Further, the hbase unit comprises a Coprocessor module and a Filter module, and both the Coprocessor module and the Filter module are used for generating an index table for data; the coprocessors module is also used for writing index data into the index table in parallel in a reverse index mode according to the index definition so as to establish a plurality of secondary indexes;

the Master unit is also used for dynamically calculating the cost of using the index according to the query condition; the coprocessors module is also used for firstly inquiring the index table according to the index definition and the inquiry condition and parallelly inquiring the data table again through the inquiry result of the index table.

Compared with the prior art, the distributed New SQL database system disclosed by the invention receives a user request sent to the Master unit by a user through the JDBC/ODBC interface unit, accesses the user request through the Master unit through the JDBC/ODBC interface unit, coordinates data communication among a plurality of processors and manages the whole flow, and preferentially sends the user request to the SQLPLanner unit; analyzing the user request through an SQLPLaner unit, and compiling and customizing an execution plan according to the user request; the technical scheme that the plan is executed in parallel through the Worker unit to obtain data, the data are merged and collected to be returned to the Master unit, and the result returned by the Master unit is returned to the user through the JDBC/ODBC interface unit solves the problems that in the prior art, the user needs to access the HBase through API programming, the use cost of complex application is too high, and the HBase cannot be used by a plurality of complex applications, obtains a standard JDBC interface, simplifies the ETL process and reduces the cost.

Drawings

FIG. 1 is a schematic structural diagram of a distributed NewSQL database system according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a JDBC/ODBC interface unit of a distributed NewSQL database system according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, a schematic structural diagram of a distributed NewSQL database system provided in embodiment 1 of the present invention is shown, where this embodiment includes:

the JDBC/ODBC interface unit 1 is used for receiving a user request sent to the Master unit 2 by a user and sending the user request to the Master unit 2; and the result returned by the Master unit 2 is returned to the user;

a Master unit 2, configured to access a user request accessed by the JDBC/ODBC interface unit 1, coordinate data communication among multiple processors, manage an overall process, and preferentially send the user request to the SQLPlaner unit 3;

the SQLPLaner unit 3 is used for analyzing the user request, compiling and customizing an execution plan according to the user request;

a Worker unit 4 for executing the plan in parallel to obtain data to be merged and summarized to be returned to the Master unit 2.

Further, the method also comprises the following steps: and the distributed transaction management unit 5 is used for coordinating multiple parties designed in the plan to complete distributed transaction management.

Usually, the SQLPlaner unit 3 finishes customizing the execution plan and returns the execution plan to the Master unit 2, and the Master unit 2 determines whether intervention of the distributed transaction management unit 5 is needed according to the content of the execution plan: if necessary, starting a distributed transaction management unit 5 to coordinate multiple parties designed in the plan to complete distributed transaction management, and then starting a Worker unit 4; if not, directly starting the Worker unit 4.

Further, the method also comprises the following steps: hbase unit 6 and SOLR unit 7, both for storing data;

the Worker unit 4 is further configured to acquire data through the hbase unit 6 and the SOLR unit 7.

Further, the method also comprises the following steps: a monitor 8, said monitor 8 being used for being responsible for metadata management and for monitoring the load of regions of said hbase unit 6, avoiding that a specific Region is overloaded too high, and reallocating regions with a coprocessors module 61 of said hbase unit 6;

the monitor 8 is connected to the Master unit 2.

Referring to fig. 2, fig. 2 is a schematic structural diagram of a JDBC/ODBC interface unit 1 of a distributed NewSQL database system in the embodiment of the present invention, where the JDBC/ODBC interface unit 1 includes:

the JDBC application program module 11 is used for carrying out interactive operation with a user and calling a JDBC object method to give an SQL statement and extract a result;

a JDBC driver manager module 12, configured to load and call a JDBC driver module 13 for the JDBC application module 11;

the JDBC driver module 13 is configured to execute the invocation of the JDBC object method, send an SQL request to the Master unit 2, and return a result returned by the Master unit 2 to the JDBC application module 11.

The embodiment keeps the common operation habit of people and provides a complete JDBC/ODBC mode interface to access the bottom layer HBase. Can be written completely in Java, and provides a JDBC driver which can be embedded in a client. The query engine will convert the SQL query into one or more HBase scans and orchestrate execution to generate a standard JDBC result set. Using the HBaseAPI, co-processor and custom filter directly, the performance is on the order of milliseconds for simple queries and seconds for millions of rows. Meanwhile, the use cost of the user is reduced, the operation is simple and convenient, and the starting speed is high. The query engine of this embodiment will not lower the execution efficiency of the SQL converted than the native API.

Further, the specific process of analyzing the user request and compiling and customizing the execution plan according to the user request by the Worker unit 4 includes:

Further, the hbase unit 6 comprises a coprocesor module 61 and a Filter module.

In this embodiment, the coprocessors module 61 and the Filter module are both configured to generate an index table for data; the coprocessors module 61 is further configured to write the index data into the index table in parallel in an inverted index manner according to the index definition, so as to establish a plurality of secondary indexes. The embodiment can realize that a plurality of secondary indexes are established when data is written through the above embodiment, so as to realize non-primary key query.

The Master unit 2 is also used for dynamically calculating the cost of using the index according to the query condition; the coprocessors module 61 is also used to query the index table according to the index definition and query conditions, and query the data table again through the query result of the index table in parallel. Since the embodiment establishes a plurality of secondary indexes when writing data, the cost of using the indexes is dynamically calculated according to the query conditions during query, the most suitable index is automatically selected, and then the data table is queried according to the index data.

Based on the structure of this embodiment, this embodiment can also implement storage and query of JSON data, specifically:

the Worker unit 4 writes JSON data as a common character string type as a whole into a data table of the hbase unit 6 as a field; triggering the coprocessors module 61 to extract data in JSON according to the field description, writing the index data into another hbase unit 6 index table in an inverted index mode, and finishing the storage of the semi-structured data; returning the index data established as a result; wherein the user request comprises storing JSON data;

or, the Worker unit 4 triggers the coprocessors module 61 to query the index table according to the query condition; the coprocessors module 61 returns the index ID of the index table to the Worker unit 4; here the coprocessors module 61 is used to increase the parallelism of the query. Then, the Worker unit 4 queries a data table according to the index ID to obtain corresponding JSON data as a result; the data table is queried using the API of hbase unit 6 and returned to Master unit 2. Wherein the user request comprises querying JSON data;

based on the structure of this embodiment, this embodiment can also implement storage and query of picture data, specifically:

the Worker unit 4 generates the picture data into an image data format encrypted by an information abstract algorithm, and writes the encrypted picture data into an original data table; specifically, the image data format after the encryption of the generated information digest algorithm is that the generated picture is MD5, that is, the picture MD5 is written into the original data table; writing the picture data into a picture data table for independent storage by using a Coprocessor module 61 in parallel, and writing the picture data as a result; wherein the user request comprises storing picture data;

or, the Worker unit 4 queries the original data table according to the query condition to obtain the image data encrypted by the information digest algorithm, namely, to obtain the picture MD 5; and inquiring a picture data table according to the encrypted image data (picture MD5) by utilizing the API of the hbase unit 6 in parallel, and acquiring picture data as a result, wherein the user request comprises inquiry picture data.

In this embodiment, an LOB type is added to the bottom layer of the hbase unit 6; the LOB type refers to implementation of BLOB type in SQL, storing large objects as a bitmap in the database, but here the LOB is implemented to build another type index for LOB type, picture data is stored as bitmap in a separate data table, and the original data table only stores index ID, thereby reducing the data table size. In the aspect of index ID generation of pictures, the method adopts MD5 calculation on picture data and takes the result of MD5 as the unique index ID of the picture data. Because the picture data can only be modified in an atomic coverage way and can be inquired independently, the retrieval speed can be greatly improved when the image data is inquired for a non-picture field.

Based on the structure of this embodiment, this embodiment can also realize storage and query of full-text data, specifically:

the Worker unit 4 writes fields needing full-text retrieval as common character string types into a data table of the hbase unit 6 for storage, then extracts index IDs according to field descriptions through the coprocessors module 61, writes index data into an index table in the SOLR unit 7, and returns the index data serving as a result to the user after completing the establishment of the index; wherein the user request comprises a field for obtaining full-text retrieval;

or, the Worker unit 4 queries a specific index ID from the SOLR unit 7 according to the query condition, the SOLR unit 7 returns the index ID according to the query condition, and queries a data table according to the index ID by using the API of the hbase unit 6 to obtain query data as a result; wherein the user request comprises a field for querying full text retrieval.

During specific implementation, a user request sent by a user to the Master unit 2 is received through the JDBC/ODBC interface unit 1, then the user request is accessed through the Master unit 2, data communication among a plurality of processors and an overall management process are coordinated, and the user request is preferentially sent to the SQLPLaner unit 3; then, the SQLPLaner unit 3 analyzes the user request, and compiles and customizes an execution plan according to the user request; and finally, the Worker unit 4 is used for executing the plan in parallel to obtain data, merging and summarizing the data to return to the Master unit 2, and further returning a result to the JDBC/ODBC interface unit 1 through the Master unit 2 to return to a user.

Compared with the prior art, the method and the device have the advantages that the user does not need to access the HBase through API programming, the use cost of the user is reduced, meanwhile, the operation is simple and convenient, and the starting speed is high.

While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.

Claims

1. A distributed NewSQL database system, comprising:

the SQLPLaner unit is used for analyzing the user request, compiling and customizing an execution plan according to the user request; specifically, whether a prestored SQL statement corresponding to the SQL statement exists in a shared cache pool or not is judged, if yes, an execution plan corresponding to the SQL statement is output, if not, syntax check is carried out on the SQL statement, if the syntax error returns error information to a user, otherwise, semantic check is carried out on the SQL statement, if the semantic error returns the error information to the user, otherwise, view and expression conversion is carried out on the SQL statement, and a corresponding conversion result is obtained; selecting an optimizer according to the conversion result to obtain a corresponding optimizer selection result; selecting a corresponding data connection mode and a corresponding connection sequence according to the selection result of the optimizer; selecting a searched path according to the connection mode and the connection sequence; generating an execution plan according to the search path, and outputting the execution plan;

the Worker unit is used for executing the execution plan in parallel to obtain data, merging and summarizing the data and returning the data to the Master unit;

the hbase unit and the SOLR unit are used for storing data; the Worker unit is further used for acquiring data through the hbase unit and the SOLR unit.

2. The distributed NewSQL database system according to claim 1, further comprising: and the distributed transaction management unit is used for coordinating the multi-party application programs designed in the plan to finish distributed transaction management.

3. The distributed NewSQL database system according to claim 1, further comprising: a monitor for taking charge of metadata management and for monitoring the load of the Region of the hbase unit, and reallocating the Region by using the Coprocessor module of the hbase unit;

the monitor is connected with the Master unit.

4. The distributed NewSQL database system of claim 1, wherein the JDBC/ODBC interface unit comprises:

5. The distributed NewSQL database system of claim 1, wherein the hbase unit comprises a Coprocessor module and a Filter module, both of which are used for generating an index table for data; the coprocessors module is also used for writing index data into the index table in parallel in a reverse index mode according to the index definition so as to establish a plurality of secondary indexes;