CN111488379B

CN111488379B - Method for optimizing Hbase large data query

Info

Publication number: CN111488379B
Application number: CN202010305095.3A
Authority: CN
Inventors: 储明
Original assignee: Focus Technology Co Ltd
Current assignee: Focus Technology Co Ltd
Priority date: 2020-04-17
Filing date: 2020-04-17
Publication date: 2022-07-19
Anticipated expiration: 2040-04-17
Also published as: CN111488379A

Abstract

The invention discloses a method for optimizing Hbase big data query, which is characterized by comprising the following steps: the method comprises the steps of storing a rowKey and an update record of Hbase in a database, storing basic information of a column in the Hbase in the database, establishing a query index based on Lucene, customizing a query view, managing a data tag and systematically querying a page. The invention aims to solve the problem that the conventional Hbase is difficult to realize efficient, convenient and complex query due to the fact that a large amount of data is stored.

Description

Method for optimizing Hbase large data query

Technical Field

The invention relates to the technical field of big data search, in particular to a method for optimizing Hbase big data query.

Background

With the development of the internet, data of internet enterprises are also exponentially increased, and the use of hadoop ecosystem to store data becomes the first choice of most enterprises, and Hbase called distributed database is also the most common storage method.

Hbase is limited by a storage principle, and can well support rowKey query, but the efficiency is very low for filtering query under complex conditions. How to optimize the Hbase large data query is a problem to be considered.

The general solution in the industry is to establish a new table in Hbase to realize secondary indexing, establish a corresponding relationship between the column data and the rowKey, and accelerate the positioning speed of the data. However, with the increase of the request complexity, the newly-built index tables are more and more, the data redundancy is more and more serious, the storage pressure is more and more, corresponding optimization schemes are available in the market, and some problems exist more or less.

Disclosure of Invention

The invention aims to solve the technical problems of improving the Hbase complex query performance, reducing the Hbase storage and calculation pressure and providing a method for optimizing Hbase large data query.

In order to solve the technical problems, the invention provides a method for optimizing Hbase large data query, which is characterized by comprising the following steps: the method comprises the steps of storing a rowKey and an update record of Hbase in a database, storing basic information listed in the Hbase in the database, establishing a query index based on Lucene, customizing a query view, managing a data tag and systematically querying a page, and specifically comprises the following steps:

s1, the database stores the rowKey and the update record of Hbase:

before data is added or deleted in Hbase, the data rowKey and related operation information are stored in a database, and the content comprises the following fields: the rowKey, the Hbase table name, the logic deletion mark, the adding time and the updating time are used for providing basic data support for establishing the lucene query index;

when the query condition column data in Hbase is changed, writing back the pipeline record information to the database, wherein the pipeline record information comprises the following fields: the rowKey, the Hbase table name, the deletion of the mark and the addition time are used for providing basic data support for updating or deleting the lucene index increment;

s2, storing the basic information listed in Hbase in the database: storing a definition relation in a database, wherein the definition relation comprises a table name, a column cluster and a column name of Hbase data, and maintaining a corresponding Chinese paraphrase for inquiring page display;

s3, establishing a query index based on Lucene: the application of the terminal day takes out corresponding data from Hbase through rowKey and update record in S1 and basic information listed in S2, and submits the corresponding data to a search engine update index based on lucene, and the search engine provides a query interface similar to an sql statement mode for the outside;

s4, customizing a query view: the user-defined query view comprises two parts of view definition and condition configuration, wherein the view definition is used for confirming a field range of a query, the condition configuration is used for restricting a data range of the query, and the view and the condition are in one-to-many relation and are stored in a database; the user-defined query view is configured in a sql-like statement mode and is consistent with a query interface of a search engine in the S3;

s5, data tag management: marking the inquired result data with user-defined labels in batches, and storing the relation in a database; the data marked with the labels can be directly filtered and inquired through the labels, or the data marked with the labels and the inquiry conditions are combined to obtain an inquiry result;

s6, systematized query page: and establishing a web service application, integrating and displaying the data in the S1, the S2 and the S3 and the functions in the S4 and the S5 to form a systematized inquiry function page.

In the S1, the database stores all the rowKey information of Hbase, records the adding and deleting time of the rowKey, and is used for date-end application to find out the added and deleted data and submit the data to the lucene index; recording an updating flow record of Lucene index related data in Hbase, and searching out updating data for a day end application and submitting the updating data to the Lucene index;

in S2, configuration information of a table and a column related to Hbase is stored in the database, and authority control and range check are performed on all data entering the Hbase according to the configuration information.

In the S3, in establishing the query index based on Lucene, after the end-of-day application receives the submitted data, processing the logic is started, which specifically includes the following steps:

s3-1: in a database hbase _ rowkey _ info table, inquiring whether a corresponding rowkey exists according to rules, and if so, returning to the rowkey; if not, generating rowkey according to rules and adding the rowkey to the hbase _ rowkey _ info table;

s3-2: for submitted data fields, performing write permission verification in an hbase _ column _ info table, wherein the fields configured in advance can be used for data submission;

s3-3: combining the acquired rowkey and the submitted data, and writing the data into an Hbase corresponding table;

s3-4: after the data is successfully written into the hbase, if the updated data relates to fields required by the lucene query, performing stream recording in a database hbase _ update _ record table so as to be convenient for incrementally acquiring the data to be updated into the lucene index;

s3-5: acquiring newly added rowkey records from a database hbase _ rowkey _ info table at a fixed time at the end of day, and acquiring updated or deleted rowkey records from an hbase _ update _ record table;

s3-6: acquiring field information used by lucene query from database hbase _ column _ info, and distinguishing query fields from data fields;

s3-7: inquiring data needing to be submitted to the lucene for index updating from the hbase by combining the information acquired by the S3-5 and the S3-6; and submitting the data to be updated to Lucene for index updating.

The invention achieves the following beneficial effects:

(1) the invention establishes a Lucene-based query index, and improves the Hbase complex condition query performance;

(2) the invention avoids Hbase from directly carrying out complex query, reduces the storage and calculation pressure of Hbase, and indirectly improves the query performance of rowKey;

(3) the invention supports the label management query function, and greatly enriches the complex query scenes;

(4) the invention provides a set of query interface with complete functions, and user experience is optimized.

Drawings

FIG. 1 is a schematic flow chart of a method in an exemplary embodiment of the invention;

FIG. 2 is a diagram illustrating the detailed steps of writing an index to data in an exemplary embodiment of the invention;

FIG. 3 is a schematic view of a query page overview in an exemplary embodiment of the invention;

FIG. 4 is a schematic diagram of a query page-view definition module in an exemplary embodiment of the invention;

FIG. 5 is a diagram of a query page-field presentation module in an exemplary embodiment of the invention;

FIG. 6 is a schematic diagram of a query page-tag management module in an exemplary embodiment of the invention;

FIG. 7 is a schematic diagram of a query page-index data presentation module in an exemplary embodiment of the invention;

FIG. 8 is a schematic diagram of a query page-Hbase data presentation module in an exemplary embodiment of the invention;

Detailed Description

The invention will be further described with reference to the drawings and the exemplary embodiments:

fig. 1 shows a schematic diagram of data write query in the exemplary embodiment of the present invention, and the specific flow is described as follows:

starting from data write Hbase;

s1, the database stores the rowKey and the update record of Hbase:

before data is added or deleted in Hbase, storing the data rowKey and the related operation information in a database to enable the data volume to be visual and controllable; the content includes the following fields: the rowKey, the Hbase table name, the logic deletion mark, the adding time and the updating time are used for providing basic data support for establishing the lucene query index;

when the data of the query condition column in Hbase is changed, writing back the pipeline record information to the database, wherein the pipeline record information comprises the following fields: the rowKey, the Hbase table name, the deletion of the mark and the addition time are used for providing basic data support for updating or deleting the lucene index increment; the date terminal application can conveniently find out the updated data and submit the updated data to the lucene index; thereby ensuring the real-time performance and data integrity of the index query;

s2, storing the basic information listed in Hbase in the database: storing a definition relation in a database, wherein the definition relation comprises a table name, a column cluster and a column name of Hbase data, and maintaining a corresponding Chinese paraphrase for inquiring page display; configuration information of a relevant table and a relevant list of Hbase is stored in a database, authority control and range verification are carried out on all data entering the Hbase according to the configuration information, the data in the Hbase becomes more transparent, the range is controllable, and basic support is provided for optimizing a query method;

s3, establishing a query index based on Lucene: the terminal application takes out corresponding data from Hbase through RowKey addition and deletion record and updating stream record in S1 and column related configuration information in S2, submits the corresponding data to a search engine based on lucene to update indexes, is simple and efficient, only extracts data related to query conditions and submits the data to the indexes, and reduces the pressure of the search engine; the search engine provides an external query interface in a sql-like statement mode, so that the user request can be better identified, and the search engine is flexible and convenient;

s4, customizing a query view: the user-defined query view comprises a view definition part and a condition configuration part, wherein the view definition part is used for confirming a field range of query, the condition configuration part is used for restricting a data range of query, and the view and the condition are in one-to-many relationship and are stored in a database; the user-defined query view is configured in a sql-like statement mode and is consistent with a query interface of a search engine in the S3; the view definition mode with separated query fields and query conditions is provided, a user can store common query fields and table information into a view in a sql-like mode, and the common query conditions are stored into condition configuration in the sql-like mode on the basis of the view, so that the user can conveniently and efficiently splice and modify query requests;

s5, data tag management: marking the inquired result data with user-defined labels in batches, and storing the relation in a database; the data marked with the labels can be directly filtered and inquired through the labels, or the data marked with the labels and the inquiry conditions are combined to obtain an inquiry result; the data tag mechanism is provided, a user can print custom tags in batch according to specified data, a plurality of tags can be combined for use, and the tags and the query view can also be combined for use, so that the user can be helped to quickly locate the data to be queried, and the data tag mechanism is flexible and convenient;

s6, systematizing the inquiry page: and a web service application is established, and the data of S1, S2 and S3 and the functions of S4 and S5 are integrated and displayed to form a systematized inquiry function page, so that the operation of a user is facilitated. All data and functions are integrated on a systematized function page, so that a user can quickly and continuously complete all query operations, page switching times and click times are reduced, and user experience is improved;

at this point, the data flow ends from writing to querying.

Fig. 2 is a schematic diagram illustrating detailed steps of data writing into an index in an exemplary embodiment of the present invention, where fig. 2 is developed based on the data writing flow in fig. 1, and arrows in fig. 2 indicate a sequence of steps, which are specifically described as follows:

beginning: after receiving the submitted data, the application starts processing logic;

p1: after the data is taken, inquiring whether the corresponding rowkey exists in a database hbase _ rowkey _ info table according to rules, and if so, returning to the rowkey; if not, generating rowkey according to rules and adding the rowkey to the hbase _ rowkey _ info table;

p2: for submitted data fields, write permission verification is carried out in the hbase _ column _ info table, and data submission can be carried out only through fields configured in advance so as to ensure the controllability of data and the convenience of query;

p3: combining the acquired rowkey and the submitted data, and writing the data into an Hbase corresponding table;

p4: after the data is successfully written into the hbase, if the updated data relates to fields required by the lucene query, performing stream recording in a database hbase _ update _ record table so as to be convenient for incrementally acquiring the data to be updated into the lucene index;

p5: acquiring a newly added rowkey record from a database hbase _ rowkey _ info table at regular time by applying, and acquiring an updated/deleted rowkey record from a hbase _ update _ record table;

p6: field information used for lucene query is obtained from a database hbase _ column _ info, and a query field and a data field are distinguished, mainly because the amount of the data fields used for storage is often multiple times of the query condition, and the cost of all indexes built is high, so that the data field is abandoned, and resources and pressure are reduced;

p7: inquiring data needing to be submitted to lucene for index updating from the hbase by combining the information obtained from P5 and P6;

and (4) ending: after the data to be updated is taken, submitting the data to Lucene for index updating;

fig. 3 is a schematic view of an overview of a query page in an exemplary embodiment of the present invention, which is a query function interface formed on the basis of the query data flow of fig. 1, and fig. 4 to 8 are detailed schematic views of an interface module of fig. 4;

the view definition module shown in fig. 4 clicks on the "create view" button, configures the view definition through the syntax of the class sql and stores the view definition to confirm which fields are to be queried, as exemplified in the figure: "select c.pid, p.generator, p.telephone, p.name, p.email, p.mobile active flag, p.mail flag, p.county from common | c # personal | p", where "|" represents an alias and "#" represents an association table query; if the preset view definition exists, the preset view definition can be selected in the pull-down frame directly; in the query condition input box, the condition of the query is input, as exemplified in the figure: "pid is 1or generator is 0", since here is the query lucene index, no alias needs to be specified; if the query condition is required to be stored, clicking a 'storage' button above a condition input box; if the preset query condition exists, clicking a history button above the condition input box to browse and select; and clicking a query button above the condition input box to display the related data queried by the index in the module D.

As shown in fig. 5, after the view definition is checked, the background queries information of relevant fields from the database according to the view definition, and performs pagination display in a table form; if there is a discrepancy between the data field desired to be viewed and the view definition, the check-in of the presentation field can be done manually, with the default being a full selection.

As shown in fig. 6, the tag management module selects an existing data tag in a drop-down input box, which illustrates: the 'demonstration tag 1' is clicked, a 'tag filtering' button on the right side of the drop-down input box is clicked, the secondary filtering query of the lucene index is triggered by combining the search condition of the module A, and the result is displayed in an index data display module (shown in figure 8); selecting an existing data tag or inputting a self-defined tag name in a pull-down input box, checking a rowKey record of Hbase needing tagging in an index data display module (figure 8), and clicking a 'tag saving' button on the right side of the pull-down input box of a tag management module (figure 7) to save the corresponding relation between the data tag and the rowKey of the Hbase; and if the data label does not exist, automatically creating and saving the data label to the database.

As shown in fig. 7, after the lucene index query is triggered, rowKey (uid) of Hbase, fields related to query conditions, and data tags are displayed in pages in the form of a table, where hyperlink is attached to rowKey, after clicking, the query of Hbase can be triggered, and the result is displayed in the E module.

As for the Hbase data display module shown in fig. 8, after clicking the hyperlink of rowKey in the index data display module (fig. 8), the background queries Hbase, and then performs paging display in the form of a table; so far, the query process ends.

The invention is mainly used for providing a method and a system for optimizing Hbase big data query, and has the beneficial effects that:

(2) the invention avoids Hbase from directly carrying out complex query, reduces the storage and calculation pressure of Hbase, and indirectly improves the performance of rowKey query;

(4) the invention provides a set of systematic query interface with complete functions, simplifies the query operation process and optimizes the user experience.

The above embodiments do not limit the present invention in any way, and all other modifications and applications that can be made to the above embodiments in equivalent ways are within the scope of the present invention.

Claims

1. A method for optimizing Hbase large data query is characterized by comprising the following steps: the method comprises the steps of storing a rowKey and an update record of Hbase in a database, storing basic information listed in the Hbase in the database, establishing a query index based on Lucene, customizing a query view, managing a data tag and systematically querying a page, and specifically comprises the following steps:

s1, the database stores the rowKey and the update record of Hbase:

s5, data tag management: printing user-defined labels on the inquired result data in batches, and storing the relation in a database; the data marked with the label is directly filtered and inquired through the label, or the data is combined with the inquiry condition to form an inquiry result;

2. The method of claim 1, wherein the method comprises the following steps: in the S1, the database stores all the rowKey information of Hbase, records the adding and deleting time of the rowKey, and is used for date-end application to find out the added and deleted data and submit the data to the lucene index; and recording an updating flow record of the data related to the Lucene index in the Hbase, and finding out the updating data for the end-of-day application and submitting the updating data to the Lucene index.

3. The method of claim 2, wherein the method comprises the following steps: in S2, the database stores the configuration information of the Hbase related table and column, and performs authority control and range check on all data entering the Hbase according to the configuration information.

4. The method of claim 3, wherein the Hbase big data query is optimized by: in S3, in the establishing of the query index based on Lucene, after the end-of-day application receives the submitted data, starting processing logic, specifically including the following steps:

s3-2: for submitted data fields, performing write permission verification in an hbase _ column _ info table, and submitting data in preconfigured fields;

s3-6: acquiring field information used by lucene query from a database hbase _ column _ info, and distinguishing a query field from a data field;