CN108241724A - A kind of metadata management method and device - Google Patents
A kind of metadata management method and device Download PDFInfo
- Publication number
- CN108241724A CN108241724A CN201710331521.9A CN201710331521A CN108241724A CN 108241724 A CN108241724 A CN 108241724A CN 201710331521 A CN201710331521 A CN 201710331521A CN 108241724 A CN108241724 A CN 108241724A
- Authority
- CN
- China
- Prior art keywords
- sentences
- operational order
- name
- column family
- row
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Abstract
The application provides a kind of metadata management method and device, and this method is applied to the Master node devices in HBase frameworks, and this method is：When detecting the operational order for HBase table, parse and judge whether the operational order is changed the table structure of HBase table；Change includes increasing table, deletes table, increases column family, deletes column family, increases row and delete at least one of row；When determining that the operational order changes the table structure of HBase table, the table structural metadata of change is synchronized in the bivariate table pre-established；The table structural metadata of change includes the table name, column family name and the row name that increase or delete.Automatic collection and storage the method achieve HBase metadata this kind of to table structure, this kind of metadata of the user based on storage can be quickly obtained the table structure of HBase table, without removing whole records of traversal HBase table again.
Description
Technical field
This application involves database technical field more particularly to a kind of metadata management methods and device.
Background technology
Hadoop databases (Hadoop Database, Hbase) are a kind of the distributed, telescopic non-of column storage
Relationship type (NoSQL, Not Only SQL) database, it is built in Hadoop infrastructure.With traditional relational data
Library is the same, and the table (Table) of HBase is made of, but HBase same rows can store the value of different moments row and column, together
When multiple row can form a column family (Column Family).So data model defined in HBase and traditional relational
The data model (row and column form two dimensional model) of database has bigger difference, it effectively define one it is four-dimensional
Data model.
Metadata (Meta Data) is the data about data, and in Database Systems, metadata is in descriptive data base
The structure of data and the data of method for building up, contribute to the developer of database administrator and database to be able to conveniently find them
Data of concern.
HBase is only capable of administrative section metadata at present, this limits using and being based on for HBase to a certain extent
The application and development of HBase.
Invention content
In view of this, the application provides a kind of metadata management method and device, to realize the table structure to HBase table
The acquisition and storage of metadata enhance the availability of HBase.
Specifically, the application is achieved by the following technical solution：
For the application in a first aspect, providing a kind of metadata management method, the method is applied to the master in HBase frameworks
Master node devices, the method includes：
When detecting the operational order for HBase table, the operational order is parsed, and judge that the operational order is
The no table structure to HBase table is changed；The change includes increasing table, deletes table, increase column family, delete column family, increase
At least one of row and deletion row；
It is when determining that the operational order changes the table structure of HBase table, the table structural metadata of change is same
It walks in the bivariate table pre-established；The table structural metadata of the change includes the table name, column family name and the row that increase or delete
Name.
The application second aspect provides a kind of meta data management device, applied to the main Master sections in HBase frameworks
Point device has the function of to realize the above method.The function can also be performed corresponding by hardware realization by hardware
Software realize.The hardware or software include one or more and the corresponding module of above-mentioned function or unit.
In a kind of possible realization method, described device includes：
Resolution unit is instructed, for when detecting the operational order for HBase table, parsing the operational order, and
Judge whether the operational order is changed the table structure of HBase table；The change includes increasing table, deletes table, increases
Column family deletes column family, increases row and deletes at least one of row；
Metadata acquisition unit, for determining table knot of the operational order to HBase table in described instruction resolution unit
When structure is changed, the table structural metadata for the HBase table that the operational order is related to is synchronized to the bivariate table pre-established
In；The table structural metadata of the change includes the table name, column family name and the row name that increase or delete.
In alternatively possible realization, described device can include processor, memory and bus, the processor and institute
It states and is connected with each other by bus system between memory；The processor is referred to by reading the logic stored in the memory
It enables, performs the metadata management method shown in the application first aspect.
As can be seen from the above technical solutions, it in the application, realizes to this kind of HBase members number of the table structure of HBase table
According to automatic collection and storage, this kind of HBase metadata of the user based on storage can be quickly obtained the table structure of HBase table,
Without the whole for traversing the HBase table is gone to record again.The application promoted the table structural metadata of HBase table from can not manage to
The change that can be managed.
Description of the drawings
Fig. 1 is the table structure diagram of HBase；
Fig. 2 is the configuration diagram for the HBase that the application provides；
Fig. 3 is the method flow diagram that the application provides；
Fig. 4 is the configuration diagram for the web application that the application provides；
Fig. 5 is the apparatus function module frame chart that the application provides；
Fig. 6 is the hardware architecture diagram for Fig. 5 shown devices that the application provides.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, it is right in the following with reference to the drawings and specific embodiments
The present invention is described in detail.
The table structure of HBase may refer to shown in Fig. 1, at least have the characteristics that：1) all there are one can for the often row of table
The line unit of sequence and arbitrary more row, row can dynamically increase as needed, and different rows can be different in same table
Row；2) data in each unit of table can have multiple versions, and version number distributes, such as can be automatically under default situations
Timestamp when cell is inserted into；3) data type is single, i.e., the data in table are all character strings, without type；4) rule of table
Mould is big, and a table can have billions of rows, row up to a million.
HBase can be preferably used using metadata, HBase can support the management to part metadata at present, than
Such as to the management of the metadata of dial gauge data storage location.But also there is the metadata of part, HBase can't provide unified
Management means, such as HBase wouldn't support the business description to table data and the management of this kind of HBase metadata of table structure, this
Problems with may be brought：
First, there can be many row in a line since HBase table supports not only, and support to have between going different
Row, if so to obtain all row names that a certain table includes needs to traverse whole rows of the table, when the scale of table is bigger
When, the complete table structures of HBase can not be got easily (including table, column family and row)；
Second, since the table of HBase only supports this data type of character string, so can not be added in the table of HBase
Annotation, in the case where the title of table or row is more ambiguous, such as when arranging entitled A, light sees that this row name user can not know
The business meaning that this row represents.
To solve the above problems, in this application, the coprocessor functions having in itself using HBase are realized to HBase
The automatic collection of this kind of HBase metadata of table structure of table and storage, and external interface is provided and is collected to HBase table and the industry arranged
Business descriptive metadata, has promoted this two classes HBase metadata from can not manage the change that can be managed.
First HBase is introduced as follows.
The common framework of HBase as shown in Fig. 2, including：Client (Client) 201, main (Master) node 202, area
Domain server (Region Server) 203 and coordination service device (Zookeeper) 204.
Wherein, Client 201 is the interface that user accesses HBase.
Master nodes 202 are connect with each Region Server 203, are responsible for each Region Server
203, such as be 203 distribution regions of Region Server (Region), the load balancing of responsible Region server 203.
Master nodes 202 have corresponded to a physical equipment in practical applications, which can be an X86-based equipment,
Such as can be X86 servers.
Each Region Server 203 safeguard its assigned region (Region), handle to the defeated of these Region
Enter/export (I/O) request.
Zookeeper204 by election, ensure whenever in cluster only there are one Master nodes 202, storage is each
The position of Region Server 203 monitors its state and real-time informing in real time to Master nodes 202 and also stores
Other information needed for HBase operations.
HBase provides the function of coprocessor (Coprocessor), which contains two kinds of plug-in units, wherein
One kind is observer (Observer).Observer's coprocessor seemingly, passes through call back function (also referred to as hook letter with flip-flop type
Number, hook) can be called execution when some particular events occur.
In this application, observer's function of coprocessor in HBase is exactly utilized, is acquired by encoding
The program of the table structural metadata of HBase table, and compiling is packaged into program bag and (this program bag is known as metadata acquisition journey below
Sequence packet), then obtained metadata acquisition program bag is deployed on the Master nodes 202 for being responsible for HBase clusters, such as
Shown in Fig. 2, then it is configured so that HBase knows the presence of the metadata acquisition program bag by doing some to HBase, and then
HBase can utilize the function of coprocessor to call the metadata acquisition program bag on specific opportunity, realize the table to HBase table
The automatic collection of structural metadata.
Based on being described above, the method provided below the application is described：
Method for the application is enable to provide smoothly is implemented, and can also be done before the method for performing the application offer following
Deployment：
First, the metadata acquisition program bag that the application provides is uploaded on the Master nodes of HBase clusters.As above
Described, metadata acquisition program bag here is one obtained by observer's coprocessor functions of HBase by adjusting back
The program that is used to acquire the table structural metadata of HBase table of the function for entrance.
Secondly, HBase is configured, so that HBase knows the presence of the metadata acquisition program bag, HBase can be passed through at that time
Coprocessor functions call the additional metadata acquisition program bag.Two kinds can be simply lifted here there are many configuration mode：
Mode 1：It is realized by the way that the core configuration file of HBase is configured, the metadata acquisition program bag under this mode
It can be loaded by all tables.Core configuration file mentioned here includes the hbase-site.xml files of HBase.
Mode 2：It is realized by table descriptor, the metadata acquisition program bag can be only to specified table under this mode
Loading.
After the completion of deployment, the Master node devices in HBase frameworks can refer to receiving the operation for HBase table
When enabling, call the metadata acquisition data packet by the coprocessor functions of itself and perform the metadata acquisition program bag
Code, to realize method flow shown in Fig. 3, referring to Fig. 3, which may include following steps：
Step 301：When detecting the operational order for HBase table, the Master node devices parsing operation refers to
Enable, and judge whether the operational order is changed the table structure of HBase table, the change include increase table, deletion table,
Increase column family, delete column family, increase row and delete at least one of row；If so, perform step 302；Otherwise, step is performed
303。
Here operational order can include data manipulation language (Data Manipulation Language, DML) and grasp
It instructs, Data Definition Language (Data Definition Language, DDL) operational order etc..
Wherein, for the DDL operational orders of HBase, can include：It establishes database table, check all tables, knot of tabling look-up
Structure deletes column family, deletes the operations such as table.
About the DML operational orders for HBase, can include：Addition record checks record, checks record in table
The operations such as a table, all records for checking some column family are deleted in sum, deletion record.
Based on the above-mentioned description to DDL operational orders and DML operational orders, where it is possible to the behaviour of the structure of table can be influenced
Work includes：It establishes database table, delete column family, deletion table, addition record etc..HBase wouldn't be supported to delete and be arranged at present, be examined certainly
Consider the development speed of HBase, perhaps delete the operation of row can realize in future.
Based on this, the application can by the structure of table there is a situation where change be divided into it is following several：
1) new table is increased；
2) new column family is increased in table；
3) new row (i.e. field) are increased under the column family of table；
4) table is deleted；
5) existing column family is deleted in table；
6) existing row are deleted under the column family of table.
Master node devices can be by keyword recognition, to judge whether the operational order received is sent out the structure of table
Change is given birth to.For example, if Master node devices have matched Drop sentences (for deleting in the operational order received
Table) or Alter sentences (for deleting column family), it may be considered that the operational order has changed table structure；If Master nodes
Equipment has matched Put sentences (for increasing table, increasing column family, increase row and increase data) in the operational order received,
It may be considered that the operational order it is possible that table structure can be changed, needs further to judge whether the Put sentences are new for increasing
Table, new column family or new row, can if it is determine that the operational order has changed table structure.
Step 302：Master node devices determine aforesaid operations instruction the table structure of HBase table is changed when,
The table structural metadata for the HBase table that the operational order is related to is synchronized in established bivariate table, the table of the change
Structural metadata includes the table name, column family name and the row name that increase or delete.
The bivariate table that the application provides can be used for the table structure for reflecting HBase table in real time, can at least include following
Field：Table name, column family name and row name.Table 1 shows a kind of structure of the bivariate table, and the bivariate table provided using the application can be with
The accurate table structure of HBase table is quickly obtained, without removing whole records of traversal HBase table again.For example, by being searched in table 1
Rope " T1 ", can quickly find included under " T1 " this HBase table show A, B, C.
1 bivariate table example of table
| Table name | Column family name | Row name |
| T1 | ||
| T2 | F1 | |
| T1 | F1 | A |
| T1 | F1 | B |
| T1 | F2 | C |
The bivariate table can be established in relevant database, such as in Access, Oracle database；Alternatively, this two
Dimension table can also be established in text file, as extensible markup language (Extensible Markup Language, XML),
In the text of the forms such as EXCEL.
After change has occurred to the table structure of HBase table in the operational order for judging to receive, Master is saved in step 302
The table structural metadata of change can be synchronized in the bivariate table pre-established by point device in the following manner：
1) if operational order can be added to two dimension for increasing table using the table name of increased table as a record
In table.In practical application, usually increase HBase table with Put sentences, if the operational order received contain Put sentences and
The Put sentences are for increasing new table, then Master node devices can increase a record in bivariate table, which includes
The Put sentences indicate the table name of increased table.For example, if the instruction of certain operational order creates table " T1 ", one can be added
Record as shown in the first row in table 1.
2) if operational order can delete what is deleted comprising operational order instruction for deleting table in bivariate table
The record of the table name of table.In practical application, HBase table usually is deleted with Drop sentences, if the operational order received includes
Drop sentences, then Master node devices can be deleted from bivariate table one or more record, this one or more record packet
The table name of table that the instruction containing the Drop sentences is deleted.If for example, certain operational order instruction delete table " T1 ", can be in upper table
The record shown in the first row, the third line, fourth line and fifth line is deleted in 1.
If 3), operational order in specified table for increasing column family, can by the column family name of increased column family, should
The table name of table is added to as a record in bivariate table where column family.In practical application, usually increase row with Put sentences
Race, if the operational order received contains Put sentences and the Put sentences are for increasing new column family, Master nodes are set
Standby to increase a record in bivariate table, which includes the Put sentences and indicates the column family name of increased column family and be somebody's turn to do
The table name of the affiliated table of column family.For example, when if the instruction of certain operational order creates column family " F1 " in table " T2 ", one can be added
Record of the item as shown in the second row in table 1.
4) if operational order is used to delete the specified column family in specified table, it can be deleted in bivariate table and contain this
The record of the column family name for the column family that operational order instruction is deleted and the table name of the affiliated table of the column family.In practical application, usually use
Alter sentences delete the column family in HBase table, if the operational order received contains Alter sentences, can be from two dimension
One or more record is deleted in table, the column family name of column family which deletes comprising Alter sentences instruction
And the table name of the affiliated table of the column family.It, can be for example, if the column family " F1 " under table " T1 " is deleted in the instruction of certain operational order
The record shown in the first row, the third line and fourth line is deleted in upper table 1.
If 5) operational order be used under specified table and column family increase row, can by the row name of increased row, should
The column family name of column family is added to as a record in bivariate table where the table name of table where row name and the row name.Practical application
In, usually increase row with Put sentences, if the operational order received contains Put sentences and the Put sentences are new for increasing
Row, then Master node devices can increase a record in bivariate table, the record include the Put sentences indicate it is increased
The table name of table belonging to the column family name of column family belonging to the row names of row, the row and the row.For example, if certain instruction instruction is at table " T1 "
Column family " F1 " under increase row " A ", then can add a record as shown in the third line in table 1.
6) if operational order is used to delete the specified row under the specified column family in specified table, can be deleted in bivariate table
Except the table name of table belonging to the column family name of column family belonging to the row name for the row for containing operational order instruction deletion, the row and the row
Record.For example, if the row " A " under the column family " F1 " of table " T1 " are deleted in certain instruction instruction, third can be deleted in upper table 1
Record shown in row.HBase, which can not be known, at present has which to arrange therefore wouldn't support to delete the operation arranged in HBase table.
Further, the application provide bivariate table in addition to can include for describe HBase table table structure field
Except, the establishment of the business description field, the business description field of column family, the business description field, table of row of table can also be included
Time field, the creation time field of column family, creation time field of row etc..The business description field difference of table, column family and row
For the business meaning of annotation table, column family and row, such as the example by the following table 2, it could be aware that row name " A " represents student's surname
Name.The creation time field of table, column family and row is respectively used to the creation time of annotation table, column family and row, which can be with
It is accurate to millisecond.HBase can provide business description of the external interface collection to table, column family and row, and store into bivariate table；
And the creation time of table, column family and row can be determined by local system time, and store into bivariate table.
2 bivariate table example of table
| Table name | Column family name | Row name | The business description of row | The creation time of row |
| T1 | F1 | A | A represents student name | yyyy-MM-dd-hh-mm-ss-SSS |
| T1 | F1 | B | B represents students' genders | yyyy-MM-dd-hh-mm-ss-SSS |
| T1 | F2 | C | C represents students ' age | yyyy-MM-dd-hh-mm-ss-SSS |
Step 303：Master node devices determine aforesaid operations instruction the table structure of HBase table is not changed
When, do not make any processing.
For example, the Update sentences in sql like language are used for the update to value, the table structure of HBase table will not be generally changed,
If containing Update sentences in the operational order that Master node devices receive, it can directly terminate this to this yuan of number
According to the calling of capture program packet, without updating bivariate table.
Optionally, asynchronous mode may be used to realize method shown in Fig. 3, i.e. Master sections in Master node devices
On the one hand point device performs step 301 to step 303, the one side normal response operational order is indicated according to the operational order
Action processing HBase table.The normal operation of HBase can not be influenced in this way.
In the application, after acquiring and storing HBase metadata, the HBase metadata of storage can also be presented
And maintenance.It is a kind of by way of web application by taking bivariate table is stored in database (hereinafter referred to as metadatabase) as an example
Mode to be presented and safeguard HBase metadata is：
First in HBase clusters or it can connect a Web container is built on the server of the metadatabase, then in Web
The web application of a Java exploitation is built on container, the framework of the web application is as shown in figure 4, can include data
Access layer, business interface are with realizing layer, REST (Representational State Transfer) API (Application
Programming Interface, application programming interface) and front end view layer.The web application can pass through JDBC
The mode of (connection of Java Data Base Connectivity, Java database) is interacted with metadatabase.
For example, the data access layer of web application is from the HBase metadata that storage can be read from metadatabase；
Business interface with realizing that layer can be responsible for dispatching data access layer and REST API, by assembled HBase metadata (such as
Intuitively, metadata can be assembled into tree-like solution) it is transmitted between scheduling data access layer and REST API；Front end view layer passes through
REST API is called to obtain assembled HBase metadata, and they orderly are presented in Web page.User can be with
The interface provided by web application, such as page increase business description to specified table or row, can also be retouched according to business
State information or the relevant HBase metadata informations of row name, table name fuzzy query.
If bivariate table is stored in text file, web application can by it is existing management file form come
HBase metadata is managed, is not described herein in detail.
So far, flow shown in Fig. 3 is completed.
Can be seen that the application by flow shown in Fig. 3 is realized pair using the coprocessor functions that HBase is carried
The automatic collection of this kind of HBase metadata of table structure of HBase table and storage, this kind of HBase metadata of the user based on storage
The table structure of a certain HBase table can be quickly obtained, without the whole for traversing the table is gone to record again, has promoted this kind of HBase
Metadata is from can not manage the change that can be managed.And the application provides interface and enables user can be to specified table or row
Increase business description, increase the availability of HBase, reduce the development difficulty of the application based on HBase.
Below by a specific embodiment, illustrate how the application manages the table structural elements of HBase table by bivariate table
Data.
First, a relevant database is created as metadatabase, and a bivariate table is established in the metadatabase, is used
In the table structural metadata of storage HBase table.The bivariate table includes the fields such as the creation time of table name, column family name, row name and row.
In original state, when in HBase without any table data, which is an empty table.
At the t1 moment, Master node devices receive an operational order, and operational order instruction creates table in HBase
" T1 " creates column family " F1 " in table " T1 " and row " A " and " B " is created under column family " F1 ".According to above-described side
Method since the operational order has changed the structure of table, then can add and the operational order phase in the bivariate table established in advance
The record of pass, it is as shown in table 3 below.
Table 3
| Table name | Column family name | Row name | Creation time |
| T1 | F1 | A | t1 |
| T1 | F1 | B | t1 |
At the t2 moment, Master node devices receive an operational order, which indicates the line unit to table " T1 "
For " F1 " the column family member " A " of " 1 " and " B " difference interpolation data.According to method as described above, due to the operational order not more
Change the table structure of HBase table, therefore do not update bivariate table, bivariate table at this time is still as shown in table 3.
At the t3 moment, Master node devices receive an operational order, which indicates the line unit to table " T1 "
" F1 " column family member " A ", " B " and " C " difference interpolation data for " 2 ".The table structure that includes by comparing the operational order and
The table structure that table 3 includes, it is known that the row " C " that the operational order includes are not appeared in table 3, i.e., the operational order is substantially changed
Table structure, so as to which row " C " increased under the column family of table " T1 " " F1 " are synchronized in bivariate table, updated bivariate table
It is as shown in table 4 below.
Table 4
| Table name | Column family name | Row name | The creation time of row |
| T1 | F1 | A | t1 |
| T1 | F1 | B | t1 |
| T1 | F1 | C | t3 |
At the t4 moment, Master node devices receive an operational order, and operational order instruction is deleted under table " T1 "
Column family " F1 ".Since the operational order has changed table structure, then it is entitled " F1 " column family can be deleted in the bivariate table shown in table 4
Three record, that is, delete table 4 in three record.
The method provided above the application is described.The device provided below the application is described.
Referring to Fig. 5, Fig. 5 is a kind of functional block diagram of meta data management device that provides of the application, which can be with
Applied to the Master node devices in HBase frameworks.As shown in figure 5, the device can include with lower unit：
Resolution unit 501 is instructed, for when detecting the operational order for HBase table, parsing the operational order,
And judge whether the operational order is changed the table structure of HBase table；The change includes increasing table, deletes table, increases
Add column family, delete column family, increase row and delete at least one of row.
Metadata acquisition unit 502, for determining the operational order to HBase table in described instruction resolution unit 501
Table structure when changing, the table structural metadata of change is synchronized in the bivariate table pre-established；The table of the change
Structural metadata includes the table name, column family name and the row name that increase or delete.
Optionally, described instruction resolution unit 501 is when determining that the operational order meets following at least one condition, really
The fixed operational order changes the table structure of HBase table：1) operational order includes Drop sentences；2) operation
Instruction includes Alter sentences；3) operational order includes Put sentences, and the Put sentences are used to increasing new table, new
Column family or new row.
Optionally, the bivariate table can be located in relevant database or text file, and the bivariate table includes following
Field：Table name, column family name and row name.
The table structural metadata of change with used following steps can be synchronized to and built in advance by the metadata acquisition unit 502
In vertical bivariate table：
If 1) operational order includes Drop sentences, one or more record, institute are deleted from the bivariate table
It states one or more record and includes the table name of table that Drop sentences instruction is deleted；
If 2) operational order includes Alter sentences, one or more record, institute are deleted from the bivariate table
It states one or more record and includes the column family name of column family and the table of the affiliated table of the column family that Alter sentences instruction is deleted
Name；
If 3) operational order includes Put sentences and the Put sentences are for increasing new table, in the two dimension
Increase a record in table, which includes the table name that the Put sentences indicate increased table；
If 4) operational order includes Put sentences and the Put sentences are for increasing new column family, described two
Increase a record in dimension table, which includes the Put sentences and indicate belonging to the column family name and the column family of increased column family
The table name of table；
If 5) operational order includes Put sentences and the Put sentences are for increasing new row, in the two dimension
Increase a record, row name of the record comprising the increased row of Put sentences instruction, the column family name of column family belonging to the row in table
And the table name of table belonging to the row.
Optionally, the bivariate table can also include following at least one field：The business description of table, the business of column family are retouched
The creation time of business description, the creation time of table, the creation time of column family and row state, arranged.
Optionally, described device can also include：
Instruction execution unit, for while described instruction resolution unit 501 parses the operational order, according to described
The action processing HBase table of operational order instruction.
So far, the description of Fig. 5 shown devices is completed.
Accordingly, present invention also provides the hardware configurations of Fig. 5 shown devices.Referring to Fig. 6, Fig. 6 is what the application provided
The hardware architecture diagram of Fig. 5 shown devices, the system include：Processor 601, memory 602 and bus 603；Wherein, it handles
Device 601 and memory 602 complete mutual communication by bus 603.
Wherein, processor 601 can be a CPU, and memory 602 can be nonvolatile memory (non-
Volatile memory), and metadata management logic instruction is stored in memory 602, processor 601 can be performed and be deposited
The metadata management logic instruction stored in reservoir 602, to realize the function of HBase in above-mentioned metadata approach, referring to Fig. 3 institutes
The flow shown.
As shown in fig. 6, the hardware configuration can also include the power management that a power supply module is configured as executive device
604 and input and output (I/O) interface 605.
So far, hardware configuration description shown in fig. 6 is completed.
The foregoing is merely the preferred embodiment of the application, not limiting the application, all essences in the application
God and any modification, equivalent substitution, improvement and etc. within principle, done, should be included within the scope of the application protection.
Claims (14)
1. a kind of metadata management method, which is characterized in that the main Master nodes that the method is applied in HBase frameworks are set
It is standby, the method includes：
When detecting the operational order for HBase table, the operational order is parsed, and judge whether the operational order is right
The table structure of HBase table is changed；It is described change include increase table, delete table, increase column family, delete column family, increase arrange and
Delete at least one of row；
When determining that the operational order changes the table structure of HBase table, the table structural metadata of change is synchronized to
In the bivariate table pre-established；The table structural metadata of the change includes the table name, column family name and the row name that increase or delete.
2. the method as described in claim 1, which is characterized in that when the operational order meets following at least one condition,
Determine that the operational order changes the table structure of HBase table：
The operational order includes Drop sentences；
The operational order includes Alter sentences；
The operational order includes Put sentences, and the Put sentences are used to increase new table, new column family or new row.
3. method as claimed in claim 2, which is characterized in that the bivariate table is located at relevant database or text file
In, the bivariate table includes following field：Table name, column family name and row name.
4. method as claimed in claim 3, which is characterized in that described to determine table knot of the operational order to HBase table
When structure is changed, the table structural metadata of change is synchronized in the bivariate table pre-established, including：
If the operational order includes Drop sentences, one or more record of deletion from the bivariate table, described one
Or the table name of table that a plurality of record is deleted comprising Drop sentences instruction.
5. method as claimed in claim 3, which is characterized in that described to determine table knot of the operational order to HBase table
When structure is changed, the table structural metadata of change is synchronized in the bivariate table pre-established, including：
If the operational order includes Alter sentences, one or more record of deletion from the bivariate table, described one
Or a plurality of record includes the column family name of column family and the table name of the affiliated table of the column family of Alter sentences instruction deletion.
6. method as claimed in claim 3, which is characterized in that described to determine table knot of the operational order to HBase table
When structure is changed, the table structural metadata of change is synchronized in the bivariate table pre-established, including：
If the operational order includes Put sentences and the Put sentences are for increasing new table, increase in the bivariate table
Add a record, which includes the table name that the Put sentences indicate increased table；
If the operational order includes Put sentences and the Put sentences are for increasing new column family, in the bivariate table
Increase a record, which includes the Put sentences and indicate the column family name of increased column family and the table of the affiliated table of the column family
Name；
If the operational order includes Put sentences and the Put sentences are for increasing new row, increase in the bivariate table
Add a record, which includes the Put sentences and indicate the row name of increased row, the column family name of column family belonging to the row and should
The table name of table belonging to row.
7. method as claimed in claim 3, which is characterized in that the bivariate table further includes following at least one field：Table
Business description, the business description of column family, the business description of row, the creation time of table, the creation time of column family and row establishment when
Between.
8. a kind of meta data management device, which is characterized in that the main Master nodes that described device is applied in HBase frameworks are set
Standby, described device includes：
Resolution unit is instructed, for when detecting the operational order for HBase table, parsing the operational order, and judge
Whether the operational order is changed the table structure of HBase table；It is described change include increase table, delete table, increase column family,
Column family is deleted, increase row and deletes at least one of row；
Metadata acquisition unit, for determining that the operational order sends out the table structure of HBase table in described instruction resolution unit
During raw change, the table structural metadata of change is synchronized in the bivariate table pre-established；The table structural metadata of the change
Including table name, column family name and the row name for increasing or deleting.
9. device as claimed in claim 8, which is characterized in that described instruction resolution unit is determining the operational order satisfaction
Below during at least one condition, determine that the operational order changes the table structure of HBase table：
The operational order includes Drop sentences；
The operational order includes Alter sentences；
The operational order includes Put sentences, and the Put sentences are used to increase new table, new column family or new row.
10. device as claimed in claim 9, which is characterized in that the bivariate table is located at relevant database or text file
In, the bivariate table includes following field：Table name, column family name and row name.
11. device as claimed in claim 10, which is characterized in that the metadata acquisition unit will be changed by following steps
Table structural metadata be synchronized in the bivariate table pre-established：
If the operational order includes Drop sentences, one or more record of deletion from the bivariate table, described one
Or the table name of table that a plurality of record is deleted comprising Drop sentences instruction.
12. device as claimed in claim 10, which is characterized in that the metadata acquisition unit will be changed by following steps
Table structural metadata be synchronized in the bivariate table pre-established：
If the operational order includes Alter sentences, one or more record of deletion from the bivariate table, described one
Or a plurality of record includes the column family name of column family and the table name of the affiliated table of the column family of Alter sentences instruction deletion.
13. device as claimed in claim 10, which is characterized in that the metadata acquisition unit will be changed by following steps
Table structural metadata be synchronized in the bivariate table pre-established：
If the operational order includes Put sentences and the Put sentences are for increasing new table, increase in the bivariate table
Add a record, which includes the table name that the Put sentences indicate increased table；
If the operational order includes Put sentences and the Put sentences are for increasing new column family, in the bivariate table
Increase a record, which includes the column family name and the corresponding table name of the column family that the Put sentences indicate increased column family；
If the operational order includes Put sentences and the Put sentences are for increasing new row, increase in the bivariate table
Add a record, which includes the Put sentences and indicate the row name of increased row, the column family name of column family belonging to the row and should
The table name of table belonging to row.
14. device as claimed in claim 10, which is characterized in that the bivariate table further includes following at least one field：Table
Business description, the business description of column family, the business description of row, the establishment of the creation time of table, the creation time of column family and row
Time.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710331521.9A CN108241724A (en) | 2017-05-11 | 2017-05-11 | A kind of metadata management method and device |
| PCT/CN2018/086398 WO2018205981A1 (en) | 2017-05-11 | 2018-05-10 | Metadata management |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710331521.9A CN108241724A (en) | 2017-05-11 | 2017-05-11 | A kind of metadata management method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108241724A true CN108241724A (en) | 2018-07-03 |
Family
ID=62702928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710331521.9A Pending CN108241724A (en) | 2017-05-11 | 2017-05-11 | A kind of metadata management method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108241724A (en) |
| WO (1) | WO2018205981A1 (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109189783A (en) * | 2018-08-03 | 2019-01-11 | 北京涛思数据科技有限公司 | A kind of time series data database table structure change processing method |
| CN110471927A (en) * | 2019-08-20 | 2019-11-19 | 浙江大搜车软件技术有限公司 | Metadata acquisition method, apparatus, computer equipment and storage medium |
| CN111159272A (en) * | 2019-12-31 | 2020-05-15 | 青梧桐有限责任公司 | Data quality monitoring and early warning method and system based on data warehouse and ETL |
| CN111159161A (en) * | 2019-12-31 | 2020-05-15 | 青梧桐有限责任公司 | ETL rule-based data quality monitoring and early warning system and method |
| CN113467774A (en) * | 2021-07-30 | 2021-10-01 | 北京鼎普科技股份有限公司 | WEB terminal business software development framework and method |
| CN115062084A (en) * | 2022-08-19 | 2022-09-16 | 中关村科学城城市大脑股份有限公司 | Method and device for constructing API (application program interface) based on database metadata |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110188118B (en) * | 2019-04-25 | 2021-06-25 | 广州至真信息科技有限公司 | Data synchronization method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346447A (en) * | 2014-10-28 | 2015-02-11 | 浪潮电子信息产业股份有限公司 | Partitioned connection method oriented to mixed type big data processing systems |
| CN104391957A (en) * | 2014-12-01 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Data interaction analysis method for hybrid big data processing system |
| CN105677826A (en) * | 2016-01-04 | 2016-06-15 | 博康智能网络科技股份有限公司 | Resource management method for massive unstructured data |
| CN106503243A (en) * | 2016-11-08 | 2017-03-15 | 国网山东省电力公司电力科学研究院 | Electric power big data querying method and system based on HBase secondary indexs |
| CN106503276A (en) * | 2017-01-06 | 2017-03-15 | 山东浪潮云服务信息科技有限公司 | A kind of method and apparatus of the time series databases for real-time monitoring system |
| CN106528674A (en) * | 2016-10-31 | 2017-03-22 | 厦门服云信息科技有限公司 | Method and device for high-performance query based on Hbase row keys |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853714B (en) * | 2012-11-28 | 2017-06-20 | 中国移动通信集团河南有限公司 | A kind of data processing method and device |
| CN103312791B (en) * | 2013-05-24 | 2016-02-24 | 上海和伍信息技术有限公司 | Internet of Things isomeric data storage means and system |
-
2017
- 2017-05-11 CN CN201710331521.9A patent/CN108241724A/en active Pending
-
2018
- 2018-05-10 WO PCT/CN2018/086398 patent/WO2018205981A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346447A (en) * | 2014-10-28 | 2015-02-11 | 浪潮电子信息产业股份有限公司 | Partitioned connection method oriented to mixed type big data processing systems |
| CN104391957A (en) * | 2014-12-01 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Data interaction analysis method for hybrid big data processing system |
| CN105677826A (en) * | 2016-01-04 | 2016-06-15 | 博康智能网络科技股份有限公司 | Resource management method for massive unstructured data |
| CN106528674A (en) * | 2016-10-31 | 2017-03-22 | 厦门服云信息科技有限公司 | Method and device for high-performance query based on Hbase row keys |
| CN106503243A (en) * | 2016-11-08 | 2017-03-15 | 国网山东省电力公司电力科学研究院 | Electric power big data querying method and system based on HBase secondary indexs |
| CN106503276A (en) * | 2017-01-06 | 2017-03-15 | 山东浪潮云服务信息科技有限公司 | A kind of method and apparatus of the time series databases for real-time monitoring system |
Non-Patent Citations (1)
| Title |
|---|
| HIMANSHU VASHISHTHA等: ""Enhancing Query Support in HBase via an Extended Coprocessors Framework"", 《PROCEEDINGS OF THE 4TH EUROPEAN CONFERENCE ON TOWARDS A SERVICE-BASED INTERNET》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109189783A (en) * | 2018-08-03 | 2019-01-11 | 北京涛思数据科技有限公司 | A kind of time series data database table structure change processing method |
| CN109189783B (en) * | 2018-08-03 | 2023-10-03 | 北京涛思数据科技有限公司 | Time sequence database table structure change processing method |
| CN110471927A (en) * | 2019-08-20 | 2019-11-19 | 浙江大搜车软件技术有限公司 | Metadata acquisition method, apparatus, computer equipment and storage medium |
| CN111159272A (en) * | 2019-12-31 | 2020-05-15 | 青梧桐有限责任公司 | Data quality monitoring and early warning method and system based on data warehouse and ETL |
| CN111159161A (en) * | 2019-12-31 | 2020-05-15 | 青梧桐有限责任公司 | ETL rule-based data quality monitoring and early warning system and method |
| CN113467774A (en) * | 2021-07-30 | 2021-10-01 | 北京鼎普科技股份有限公司 | WEB terminal business software development framework and method |
| CN113467774B (en) * | 2021-07-30 | 2024-01-30 | 北京鼎普科技股份有限公司 | WEB terminal business software development framework and method |
| CN115062084A (en) * | 2022-08-19 | 2022-09-16 | 中关村科学城城市大脑股份有限公司 | Method and device for constructing API (application program interface) based on database metadata |
| CN115062084B (en) * | 2022-08-19 | 2022-11-04 | 中关村科学城城市大脑股份有限公司 | Method and device for constructing API (application programming interface) based on database metadata |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018205981A1 (en) | 2018-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108241724A (en) | A kind of metadata management method and device | |
| AU2019219824B2 (en) | System for synchronization of changes in edited websites and interactive applications | |
| US20220405277A1 (en) | Joining large database tables | |
| US10268645B2 (en) | In-database provisioning of data | |
| CN104462185B (en) | A kind of digital library's cloud storage system based on mixed structure | |
| CN104067216A (en) | System and method for implementing a scalable data storage service | |
| CN106294695A (en) | A kind of implementation method towards the biggest data search engine | |
| JP2010524060A (en) | Data merging in distributed computing | |
| CN107103448A (en) | Data integrated system based on workflow | |
| WO2021032146A1 (en) | Metadata management method and apparatus, device, and storage medium | |
| US8209297B2 (en) | Data processing device and method | |
| US20130055235A1 (en) | Custom code innovation management | |
| Manghi et al. | The D-NET software toolkit: A framework for the realization, maintenance, and operation of aggregative infrastructures | |
| US11550785B2 (en) | Bidirectional mapping of hierarchical data to database object types | |
| US20110184956A1 (en) | Accessing digitally published content using re-indexing of search results | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| WO2024001493A1 (en) | Visual data analysis method and device | |
| CN111966692A (en) | Data processing method, medium, device and computing equipment for data warehouse | |
| CN105550351B (en) | The extemporaneous inquiry system of passenger's run-length data and method | |
| US9047391B2 (en) | Searching apparatus, searching method, and computer program product | |
| US20140304293A1 (en) | Apparatus and Method for Query Based Replication of Database | |
| CN115455006A (en) | Data processing method, data processing device, electronic device, and storage medium | |
| CN108170867A (en) | A kind of metadata service system | |
| CN112612453A (en) | RESTful service-driven JS object numbered musical notation data interchange platform | |
| CN117009327B (en) | Data processing method and device, computer equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180703 |