CN113297273B - Method and device for inquiring metadata and electronic equipment - Google Patents
Method and device for inquiring metadata and electronic equipment Download PDFInfo
- Publication number
- CN113297273B CN113297273B CN202110645140.4A CN202110645140A CN113297273B CN 113297273 B CN113297273 B CN 113297273B CN 202110645140 A CN202110645140 A CN 202110645140A CN 113297273 B CN113297273 B CN 113297273B
- Authority
- CN
- China
- Prior art keywords
- character length
- name
- metadata
- query request
- row
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000005192 partition Methods 0.000 claims abstract description 53
- 230000004044 response Effects 0.000 claims abstract description 10
- 238000013507 mapping Methods 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000004590 computer program Methods 0.000 abstract description 11
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a method for querying metadata, relates to the technical field of artificial intelligence, and particularly relates to the technical field of databases. The specific implementation scheme is as follows: in response to receiving a metadata query request, analyzing the metadata query request to obtain a plurality of partition field information associated with the metadata query request; analyzing the partition field information to obtain a grammar tree; determining a start line key and an end line key for a metadata query request according to the syntax tree; and scanning the HBase database according to the start line key and the end line key to obtain a query result aiming at the metadata query request. The present disclosure also provides an apparatus for querying metadata, an electronic device, a non-transitory computer-readable storage medium, a computer program product.
Description
Technical Field
The disclosure relates to the technical field of artificial intelligence, in particular to the technical field of databases, and specifically relates to a method and a device for inquiring metadata, electronic equipment and a storage medium.
Background
Common metadata management can be divided into centralized and distributed metadata management architectures. The centralized metadata management architecture employs a single metadata server. The distributed metadata management architecture then spreads the metadata across multiple nodes.
Disclosure of Invention
Provided are a method, apparatus, device, and storage medium for querying metadata.
According to a first aspect, there is provided a method of querying metadata, comprising: in response to receiving a metadata query request, analyzing the metadata query request to obtain a plurality of partition field information associated with the metadata query request; analyzing the partition field information to obtain a grammar tree; determining a start line key and an end line key for a metadata query request according to the syntax tree; and scanning the HBase database according to the start line key and the end line key to obtain a query result aiming at the metadata query request.
According to a second aspect, there is provided an apparatus for querying metadata, comprising: the response module is used for responding to the received metadata query request, analyzing the metadata query request and obtaining a plurality of partition field information associated with the metadata query request; the analysis module is used for analyzing the plurality of partition field information to obtain a grammar tree; the determining module is used for determining a start line key and an end line key for the metadata query request according to the grammar tree; and the scanning module is used for scanning the HBase database according to the start line key and the end line key to obtain a query result aiming at the metadata query request.
According to a third aspect, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method provided by the embodiments of the present disclosure.
According to a fourth aspect, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer as described above to perform the method provided by the embodiments of the present disclosure.
According to a fifth aspect, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method provided by the embodiments of the present disclosure.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 illustrates an exemplary system architecture suitable for a method or/and apparatus of querying metadata in accordance with embodiments of the present disclosure;
FIG. 2 is a flow chart of a method of querying metadata according to one embodiment of the present disclosure;
fig. 3A, 3B, and 3C are schematic diagrams of a first row key for a topic hierarchy, a second row key for a database hierarchy, a third row key for a table hierarchy, and a fourth row key for partition field information in the HBase database according to one embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a physical table according to one embodiment of the present disclosure;
FIG. 5 is a flowchart of the execution of a method of querying metadata according to one embodiment of the present disclosure;
FIG. 6 is a block diagram of an apparatus for querying metadata according to one embodiment of the present disclosure;
FIG. 7 illustrates a schematic block diagram of an example electronic device that may be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Common metadata management can be divided into centralized and distributed metadata management architectures. The centralized metadata management architecture adopts a single metadata server, and is simple to implement, but has the problems of single point of failure and the like. The distributed metadata management architecture distributes metadata on a plurality of nodes, so that the performance bottleneck and other problems of the metadata server are solved, the expandability of the metadata management architecture is improved, but the implementation is complex, and the problem of metadata consistency is introduced.
HBase (Hadoop Database) is a high-reliability, high-performance, column-oriented and scalable distributed storage system, and a large-scale structured storage cluster can be built on an inexpensive PC Server by utilizing the HBase technology. The storage of metadata may not be based on the structure of row keys and column families, but there are a series of problems.
FIG. 1 illustrates an exemplary system architecture suitable for a method or/and apparatus of querying metadata in accordance with embodiments of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present application may be applied to help those skilled in the art understand the technical content of the present application, and does not mean that the embodiments of the present application may not be used in other environments or scenarios.
As shown in fig. 1, the system architecture 100 includes a client 101, a metadata manager 102, and a data read-write manager 103.
The client 101 provides an interface for accessing the HBase, and maintains a corresponding cache to speed up the access of the HBase.
Metadata manager 102 stores metadata for HBase. In the case of reading data or writing data, the client 101 obtains metadata from the metadata manager 102, and then the client initiates a request for reading and writing data according to the metadata to the corresponding data read-write manager.
The data read-write manager 103 processes read-write requests of the client 101, responsible for interacting with the distributed file system (Hadoop Distributed File System, HDFS).
When the data needs to be written, the client 101 initiates a request for acquiring metadata to the metadata manager 102, and then the metadata manager 102 returns the address of the corresponding data read-write manager 103 to the client 101. The client 101 acquires the address returned by the metadata manager 102, then initiates a request to the corresponding data read-write manager 103 according to the address, and finally the data read-write manager 103 performs data read-write operation and returns the data to the client 101.
It should be understood that the number of clients, metadata managers, and data read-write managers in FIG. 1 is merely illustrative. There may be any number of clients, metadata managers, and data read-write managers, as desired for implementation.
Fig. 2 is a flow chart of a method of querying metadata according to one embodiment of the present disclosure.
As shown in fig. 2, the method 200 may include operations S210 to S240.
In response to receiving the metadata query request, the metadata query request is parsed to obtain a plurality of partition field information associated with the metadata query request in operation S210.
For example, the metadata manager may interact with a data read-write manager, which may interact with a distributed file system. Athlete data may be stored in a distributed file system, and athlete name and ability scoring rating may be used as zone fields. The metadata query request may be name= 'jim' and rating ≡97and jersey=23. Parsing the metadata query request may yield partition field information: name= 'jim', rating > 97.
In operation S220, the plurality of partition field information is parsed to obtain a syntax tree.
According to embodiments of the present disclosure, the syntax tree may include logical operation relationships between partition field information.
For example, parsing multiple partition field information name= 'jim' and rating ≡97 may result in a corresponding syntax tree.
In operation S230, a start line key and an end line key for the metadata query request are determined according to the above syntax tree.
According to the embodiment of the disclosure, according to the syntax tree, an initial start line key and an initial end line key for a metadata query request can be determined first, and then an analyzed start line key and an analyzed end line key are obtained.
For example, according to the above syntax data, an initial start line key statrowkey of name= 'jim' and rating ≡97and jersey=23 and an initial end line key endrowkey of player may be obtained, and for the above metadata query, the start line key obtained after parsing may be 6player3jim297 and the end line key may be player.
In operation S240, the HBase database is scanned according to the start line key and the end line key to obtain a query result for the metadata query request.
For example, the relevant data of 6 layer3jim297 is obtained from the start line key layer scan Hbase database, and this piece of data can be used as a query result.
By the embodiment of the disclosure, data storage can be more compact, and optimization scanning can be facilitated during inquiry.
Fig. 3A, 3B and 3C are schematic diagrams of a first row key for a topic hierarchy, a second row key for a database hierarchy, a third row key for a table hierarchy, and a fourth row key for partition field information in the HBase database according to one embodiment of the present disclosure.
As shown in fig. 3A, for each metadata, the HBase database may include a first row of keys for the theme hierarchy.
According to an embodiment of the present disclosure, the first row key includes the following fields sequentially arranged: module name character length, module name, namespace character length, namespace, belonging tier character length, belonging tier, custom naming character length, and custom naming.
For example, a first row key may be 4meta7default5topic6mark, where 4 is the module name symbol length, meta is the module name, 7 is the namespace character length, default is the namespace, 5 is the hierarchical character length, topic is the hierarchical, 6 is the custom naming character length, and mark is the custom naming.
As shown in fig. 3B, for each metadata, the HBase database may include a second row of keys for the database hierarchy and a third row of keys for the table hierarchy.
According to an embodiment of the present disclosure, the second row key and the third row key each include the following fields arranged in order: module name character length, module name, name space character length, name space, belonging level character length, belonging level, last level custom name character length, last level custom name, custom name character length, and custom name.
According to an embodiment of the present disclosure, the table includes a mapping table and a physical table, the mapping table includes data source information and storage path information; the physical table includes field information.
For example, a second row key may be 4meta7default8database6mark 2db, where 4 is the module name symbol length, meta is the module name, 7 is the namespace character length, default is the namespace, 8 is the belonging level character length, database is the belonging level, 6 is the last level custom name character length, mark is the last level custom name, 2 is the custom name character length, and db is the custom name.
For example, a third row key may be 4meta7default5 tables 2db6orders, where 4 is the module name symbol length, meta is the module name, 7 is the namespace character length, default is the namespace, 5 is the belonging hierarchy character length, table is the belonging hierarchy, 2 is the last hierarchy custom name character length, db is the last hierarchy custom name, 6 is the custom name character length and orders is the custom name.
As shown in fig. 3C, the above-mentioned field information includes non-partition field information and partition field information; wherein the HBase database further comprises a fourth row key for the partition field information.
Implementing the fourth row of keys according to the present disclosure includes the following fields in order: module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
For example, a fourth row key may be 9 parts 7default6orders10 banskball. Wherein 9 is a module name symbol length, part is a module name, 7 is a name space character length, default is a name space, 6 is a table name character length, orders is a table name, 10 is a partition field value character length, and baseball is a partition value.
FIG. 4 is a schematic diagram of a physical table according to one embodiment of the present disclosure.
As shown in fig. 4, the physical table Player 400 stores therein partition field information: name, rating, and team, non-partition field information: jersey and country.
The arrangement order of the fourth row keys rowkey in the HBase table corresponding to the physical table Player 400 in fig. 4 is a dictionary order, which is in turn:
9partition7default6player5henry2933B22；
9partition7default6player3jim2973A11；
9partition7default6player41ucy2953C33。
the topic of the physical table P1 layer 400 is a mark, and the first line key for the topic is 9part 7default5topic6mark, which can be understood as the athlete market of a sports tournament, where the athlete can sign up with other teams or be traded by the team to other teams. The subject may have multiple databases, with physical table Player 400 located in one of the databases db, with the second row key for that database db being 9part 7default8database6mark 2db, and the players in that database being players who can sign up with other teams. The database db may have a plurality of parallel surface levels, and the third row of keys for the surface level at which the physical table Player 400 is located is 9 parts 7default5table2db6orders, which may be understood as star players who may sign up with other teams. At least one mapping table and at least one physical table corresponding to the mapping table may be provided under the table hierarchy.
Fig. 5 is a flowchart of the execution of a method of querying metadata according to one embodiment of the present disclosure.
As shown in fig. 5, the execution flow 500 may include operations S501 to S508.
In operation S501, a metadata query request is parsed.
For example, a client initiates a metadata query request: select from Player where name = 'jim' and rating ≡97and jersey=23, and the metadata manager, in response to receiving the metadata query request, parses it to obtain a plurality of first partition field information: name= 'jim', rating > 97. Wherein the Player may be physical table Player 400 in fig. 4.
In operation S502, the partition field is parsed. And analyzing the first partition field information to obtain a grammar tree.
For example, the syntax tree may be expressed as: name= 'jim' and rating > 97
In operation S503, a start line key and an end line key are determined.
For example, from the above syntax tree, a start line key startrowkey for a metadata query request is determined: 9 parts 7default6 layer; and ending the row key endrow key:9 parts 7default6 plants.
In one embodiment, the start line key may be optimized according to the first partition field information, for example, the optimized start line key startrowkey is: 9 parts 7default6 layer3jim297.
In operation S504, the HBase database is scanned.
For example, the HBase database is scanned based on the start line key and the end line key to obtain a fourth line key rowkey corresponding to the physical table Player 400.
In operation S505, a fourth row of keys is determined.
For example, a fourth row of keys may be determined from the initial start row key:9 parts 7default6 layer5henry2933B22.
For example, a fourth row of keys may be determined from the optimized start row of keys: 9 parts 7default6 layer3jim2973A11. At this point, in effect, a fourth row of keys is skipped directly: the range of scanning can be reduced by using the optimized start line key of 9 parts 7default6 layer5henry 2933822.
In operation S506, the start row key is matched with the fourth row key. If the matching is successful, executing operation S507; if the match fails, operation S508 is performed.
For example, according to a fourth row of keys: the second partition field information name= jim, rating=97, team=a11 obtained by 9partition7default6 layer3jim2973a11 is matched with the first partition field information. For example, matching name= jim with name= 'jim' to obtain a matching result of ure; matching rating=97 with rating=97, and obtaining a matching result of wire. Meanwhile, if the relationship between the two pieces of first partition field information is 'and', the second partition field information of the fourth row key is matched with the first partition field information, so that 9partition7default6 layer3jim2973a11 accords with the metadata query request, and the operation jumps to operation S507.
In operation S507, a query result is obtained. The process returns to operation S505.
For example, a fourth row of keys: the 9part 7default6 layer3jim2973A11 is successfully matched with the start row key, and metadata corresponding to the row key is used as a query result. And jumps back to operation S505.
For example, after the matching operation for one fourth row key 9part 7default6 layer3jim2973a11 is completed, the matching operation is performed for the following fourth row key 9part 7default6 layer4lucy2953C 33. According to the fourth row of keys: the second partition field information name=lucy, rating=95, team=c33 obtained by 9partition7default6 layer4lucy2953C33 is matched with the first partition field information. For example, matching name=lucy with name= 'jim', resulting in a matching result of False; matching rating=95 with rating+.97, the matching result is False. Meanwhile, if the relationship between the two pieces of first partition field information is 'and', the matching of the second partition field information and the first partition field information for the fourth row key is not passed, so that the 9partition7default6 layer4 size 2953C33 does not meet the metadata query request, and the operation jumps to operation S508.
In operation S508, the end row key and the fourth row key are matched. If the matching is successful, the process is ended, if the matching is failed, the process returns to operation S505, and another fourth row key is determined.
For example, the fourth line key 9part 7default6 layer4lucy2953C33 fails to match the start line key, and matches it with the end line key, and the result obtained is also a match failure. Returning to operation S505, a fourth row of keys is determined.
By the embodiment of the disclosure, the corresponding physical table can be filtered, and an accurate query result is returned as much as possible.
Fig. 6 is a block diagram of an apparatus for querying metadata according to one embodiment of the present disclosure.
As shown in fig. 6, the apparatus for querying metadata includes a response module 610, a parsing module 620, a determining module 630, and a scanning module 640.
The response module 610 is configured to parse the metadata query request in response to receiving the metadata query request, and obtain a plurality of partition field information associated with the metadata query request.
The parsing module 620 is configured to parse the plurality of partition field information to obtain a syntax tree.
A determining module 630, configured to determine a start line key and an end line key for the metadata query request according to the syntax tree.
And the scanning module 640 is configured to scan the HBase database according to the start line key and the end line key to obtain a query result for the metadata query request.
As an alternative embodiment, the HBase database includes, for each metadata, a first row key for the topic hierarchy, a second row key for the database hierarchy, and a third row key for the table hierarchy; wherein, the table comprises a mapping table and a physical table, and the mapping table comprises data source information and storage path information; the physical table includes field information.
As an alternative embodiment, the first row key includes the following fields arranged in sequence: module name character length, module name, namespace character length, namespace, belonging tier character length, belonging tier, custom naming character length, and custom naming.
As an alternative embodiment, the second row key and the third row key each include the following fields arranged in sequence: module name character length, module name, name space character length, name space, belonging level character length, belonging level, last level custom name character length, last level custom name, custom name character length, and custom name.
As an alternative embodiment, the field information includes non-partition field information and partition field information; the HBase database further includes a fourth row key for the partition field information, where the fourth row key includes the following fields in sequence: module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
As an alternative embodiment, the scanning module includes: and the matching unit is used for respectively matching the start line key and the end line key with a fourth line key in the HBase database to obtain a query result aiming at the metadata query request.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
Fig. 7 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 7, the apparatus 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 may also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Various components in device 700 are connected to I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, etc.; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, an optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the respective methods and processes described above, for example, a method of querying metadata. For example, in some embodiments, the method of querying metadata may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 700 via ROM 702 and/or communication unit 709. When a computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the method of querying metadata described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method of method querying metadata by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the disclosure, data storage can be more compact, and optimization scanning can be facilitated during inquiry.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present application may be performed in parallel or sequentially or in a different order, provided that the desired results of the disclosed embodiments are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (12)
1. A method of querying metadata, comprising:
in response to receiving a metadata query request, resolving the metadata query request to obtain a plurality of partition field information associated with the metadata query request;
analyzing the partition field information to obtain a grammar tree;
determining a start line key and an end line key for a metadata query request according to the grammar tree; and
scanning an HBase database according to the start line key and the end line key to obtain a query result aiming at the metadata query request;
wherein, for each metadata, the HBase database comprises a first row of keys for a topic hierarchy, a second row of keys for a database hierarchy, and a third row of keys for a table hierarchy;
the table comprises a mapping table and a physical table, wherein the mapping table comprises data source information and storage path information; the physical table includes field information.
2. The method of claim 1, wherein the first row of keys comprises the following fields, arranged in order:
module name character length, module name, namespace character length, namespace, belonging tier character length, belonging tier, custom naming character length, and custom naming.
3. The method of claim 1, wherein the second and third rows of keys each comprise the following fields arranged in sequence:
module name character length, module name, name space character length, name space, belonging level character length, belonging level, last level custom name character length, last level custom name, custom name character length, and custom name.
4. The method of claim 1, wherein the field information comprises non-partitioned field information and partitioned field information;
the HBase database further comprises a fourth row key aiming at the partition field information, and the fourth row key sequentially comprises the following fields:
module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
5. The method of claim 4, wherein the scanning the HBase database according to the start row key and the end row key to obtain a query result for the metadata query request comprises:
and respectively matching the start line key and the end line key with a fourth line key in the HBase database to obtain a query result aiming at the metadata query request.
6. An apparatus for querying metadata, comprising:
the response module is used for responding to the received metadata query request, analyzing the metadata query request and obtaining a plurality of partition field information associated with the metadata query request;
the analysis module is used for analyzing the plurality of partition field information to obtain a grammar tree;
the determining module is used for determining a start line key and an end line key for the metadata query request according to the grammar tree;
the scanning module is used for scanning the HBase database according to the start line key and the end line key to obtain a query result aiming at the metadata query request;
wherein, for each metadata, the HBase database comprises a first row of keys for a topic hierarchy, a second row of keys for a database hierarchy, and a third row of keys for a table hierarchy;
the table comprises a mapping table and a physical table, wherein the mapping table comprises data source information and storage path information; the physical table includes field information.
7. The apparatus of claim 6, wherein the first row of keys comprises the following fields, arranged in order:
module name character length, module name, namespace character length, namespace, belonging tier character length, belonging tier, custom naming character length, and custom naming.
8. The apparatus of claim 6, wherein the second and third rows of keys each comprise the following fields arranged in sequence:
module name character length, module name, name space character length, name space, belonging level character length, belonging level, last level custom name character length, last level custom name, custom name character length, and custom name.
9. The apparatus of claim 6, wherein the field information comprises non-partitioned field information and partitioned field information;
the HBase database further comprises a fourth row key aiming at the partition field information, and the fourth row key sequentially comprises the following fields:
module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
10. The apparatus of claim 9, wherein the scanning module comprises:
and the matching unit is used for respectively matching the start line key and the end line key with a fourth line key in the HBase database to obtain a query result aiming at the metadata query request.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110645140.4A CN113297273B (en) | 2021-06-09 | 2021-06-09 | Method and device for inquiring metadata and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110645140.4A CN113297273B (en) | 2021-06-09 | 2021-06-09 | Method and device for inquiring metadata and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113297273A CN113297273A (en) | 2021-08-24 |
| CN113297273B true CN113297273B (en) | 2024-03-01 |
Family
ID=77327757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110645140.4A Active CN113297273B (en) | 2021-06-09 | 2021-06-09 | Method and device for inquiring metadata and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297273B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104252536A (en) * | 2014-09-16 | 2014-12-31 | 福建新大陆软件工程有限公司 | Hbase-based internet log data inquiring method and device |
| CN104778406A (en) * | 2015-04-08 | 2015-07-15 | 公安部第一研究所 | Method for uniformly naming malicious codes based on file fingerprint and system thereof |
| CN106326429A (en) * | 2016-08-25 | 2017-01-11 | 武汉光谷信息技术股份有限公司 | Hbase second-level query scheme based on solr |
| CN106326361A (en) * | 2016-08-10 | 2017-01-11 | 中国农业银行股份有限公司 | HBase database-based data inquiry method and device |
| CN106682139A (en) * | 2016-12-19 | 2017-05-17 | 深圳盒子支付信息技术有限公司 | Method and system for achieving HBase multi-condition query based on Solr |
| CN109299102A (en) * | 2018-10-23 | 2019-02-01 | 中国电子科技集团公司第二十八研究所 | A kind of HBase secondary index system and method based on Elastcisearch |
| CN109299110A (en) * | 2018-11-09 | 2019-02-01 | 东软集团股份有限公司 | Data query method, apparatus, storage medium and electronic equipment |
| CN109785034A (en) * | 2018-11-13 | 2019-05-21 | 北京码牛科技有限公司 | User's portrait generation method, device, electronic equipment and computer-readable medium |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110347722A (en) * | 2019-07-11 | 2019-10-18 | 软通智慧科技有限公司 | Data capture method, device, equipment and storage medium based on HBase |
| CN110362549A (en) * | 2019-06-17 | 2019-10-22 | 平安普惠企业管理有限公司 | Log memory search method, electronic device and computer equipment |
| CN110427366A (en) * | 2019-06-29 | 2019-11-08 | 苏州浪潮智能科技有限公司 | Index synchronous method, apparatus and system based on the phoenix transformation old system of hbase |
| CN111552753A (en) * | 2020-04-24 | 2020-08-18 | 中国科学院空天信息创新研究院 | Global massive remote sensing data organization and management method and system |
| CN112860695A (en) * | 2021-02-08 | 2021-05-28 | 北京百度网讯科技有限公司 | Monitoring data query method, device, equipment, storage medium and program product |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014082198A1 (en) * | 2012-11-27 | 2014-06-05 | Nokia Siemens Networks Oy | Multiple fields parallel query method and corresponding storage organization |
| US11615142B2 (en) * | 2018-08-20 | 2023-03-28 | Salesforce, Inc. | Mapping and query service between object oriented programming objects and deep key-value data stores |
-
2021
- 2021-06-09 CN CN202110645140.4A patent/CN113297273B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104252536A (en) * | 2014-09-16 | 2014-12-31 | 福建新大陆软件工程有限公司 | Hbase-based internet log data inquiring method and device |
| CN104778406A (en) * | 2015-04-08 | 2015-07-15 | 公安部第一研究所 | Method for uniformly naming malicious codes based on file fingerprint and system thereof |
| CN106326361A (en) * | 2016-08-10 | 2017-01-11 | 中国农业银行股份有限公司 | HBase database-based data inquiry method and device |
| CN106326429A (en) * | 2016-08-25 | 2017-01-11 | 武汉光谷信息技术股份有限公司 | Hbase second-level query scheme based on solr |
| CN106682139A (en) * | 2016-12-19 | 2017-05-17 | 深圳盒子支付信息技术有限公司 | Method and system for achieving HBase multi-condition query based on Solr |
| CN109299102A (en) * | 2018-10-23 | 2019-02-01 | 中国电子科技集团公司第二十八研究所 | A kind of HBase secondary index system and method based on Elastcisearch |
| CN109299110A (en) * | 2018-11-09 | 2019-02-01 | 东软集团股份有限公司 | Data query method, apparatus, storage medium and electronic equipment |
| CN109785034A (en) * | 2018-11-13 | 2019-05-21 | 北京码牛科技有限公司 | User's portrait generation method, device, electronic equipment and computer-readable medium |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110362549A (en) * | 2019-06-17 | 2019-10-22 | 平安普惠企业管理有限公司 | Log memory search method, electronic device and computer equipment |
| CN110427366A (en) * | 2019-06-29 | 2019-11-08 | 苏州浪潮智能科技有限公司 | Index synchronous method, apparatus and system based on the phoenix transformation old system of hbase |
| CN110347722A (en) * | 2019-07-11 | 2019-10-18 | 软通智慧科技有限公司 | Data capture method, device, equipment and storage medium based on HBase |
| CN111552753A (en) * | 2020-04-24 | 2020-08-18 | 中国科学院空天信息创新研究院 | Global massive remote sensing data organization and management method and system |
| CN112860695A (en) * | 2021-02-08 | 2021-05-28 | 北京百度网讯科技有限公司 | Monitoring data query method, device, equipment, storage medium and program product |
Non-Patent Citations (2)
| Title |
|---|
| 基于内存的HBase二级索引设计;崔晨;郑林江;韩凤萍;何牧君;;计算机应用(06);64-70 * |
| 针对微博信息分析的HBase存储结构设计;陈希林;马丁;;信息网络安全(09);275-279 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113297273A (en) | 2021-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7170082B2 (en) | Method and device for generating information, electronic device, storage medium and computer program | |
| US11334544B2 (en) | Method, apparatus, device and medium for storing and querying data | |
| US20150026189A1 (en) | Index structure for a relational database table | |
| KR20110039233A (en) | Resource locator suggestions from input character sequence | |
| US11222067B2 (en) | Multi-index method and apparatus, cloud system and computer-readable storage medium | |
| JP2022046759A (en) | Retrieval method, device, electronic apparatus and storage medium | |
| CN111401033A (en) | Event extraction method, event extraction device and electronic equipment | |
| JP7093825B2 (en) | Man-machine dialogue methods, devices, and equipment | |
| US11573961B2 (en) | Delta graph traversing system | |
| US20140379753A1 (en) | Ambiguous queries in configuration management databases | |
| CN109815240B (en) | Method, apparatus, device and storage medium for managing index | |
| JP5844895B2 (en) | Distributed data search system, distributed data search method, and management computer | |
| CN111435406A (en) | Method and device for correcting database statement spelling errors | |
| JP2022003509A (en) | Entity relation mining method, device, electronic device, computer readable storage medium, and computer program | |
| CN111259107A (en) | Storage method and device of determinant text and electronic equipment | |
| JP2021535473A (en) | Token matching in a large document corpus | |
| US20220358178A1 (en) | Data query method, electronic device, and storage medium | |
| JP2021082306A (en) | Method, apparatus, device, and computer-readable storage medium for determining target content | |
| CN113760839A (en) | Log data compression processing method and device, electronic equipment and storage medium | |
| US10049102B2 (en) | Method and system for providing semantics based technical support | |
| CN111666372A (en) | Method and device for analyzing query term query, electronic equipment and readable storage medium | |
| CN111290714B (en) | Data reading method and device | |
| US11327962B1 (en) | Real-time analytical database system for querying data of transactional systems | |
| CN113297273B (en) | Method and device for inquiring metadata and electronic equipment | |
| CN111666302A (en) | User ranking query method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |