CN113297273A - Method and device for querying metadata and electronic equipment - Google Patents
Method and device for querying metadata and electronic equipment Download PDFInfo
- Publication number
- CN113297273A CN113297273A CN202110645140.4A CN202110645140A CN113297273A CN 113297273 A CN113297273 A CN 113297273A CN 202110645140 A CN202110645140 A CN 202110645140A CN 113297273 A CN113297273 A CN 113297273A
- Authority
- CN
- China
- Prior art keywords
- character length
- metadata
- name
- hierarchy
- query request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000005192 partition Methods 0.000 claims abstract description 60
- 238000004590 computer program Methods 0.000 claims abstract description 12
- 230000004044 response Effects 0.000 claims abstract description 10
- 238000013507 mapping Methods 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 2
- 239000000126 substance Substances 0.000 claims 1
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 11
- 238000007726 management method Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 241000023320 Luma <angiosperm> Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Abstract
The present disclosure provides a method for querying metadata, which relates to the technical field of artificial intelligence, in particular to the technical field of databases. The specific implementation scheme is as follows: in response to receiving a metadata query request, analyzing the metadata query request to obtain a plurality of partition field information associated with the metadata query request; analyzing the plurality of partition field information to obtain a syntax tree; determining a start line key and an end line key aiming at the metadata query request according to the syntax tree; and scanning the HBase database according to the start row key and the end row key to obtain a query result aiming at the metadata query request. The present disclosure also provides an apparatus for querying metadata, an electronic device, a non-transitory computer-readable storage medium, and a computer program product.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence technologies, and in particular, to the field of database technologies, and in particular, to a method and an apparatus for querying metadata, an electronic device, and a storage medium.
Background
Common metadata management can be divided into centralized and distributed metadata management architectures. The centralized metadata management architecture employs a single metadata server. The distributed metadata management architecture distributes the metadata across multiple nodes.
Disclosure of Invention
A method, apparatus, device, and storage medium for querying metadata are provided.
According to a first aspect, there is provided a method of querying metadata, comprising: in response to receiving a metadata query request, analyzing the metadata query request to obtain a plurality of partition field information associated with the metadata query request; analyzing the plurality of partition field information to obtain a syntax tree; determining a start line key and an end line key aiming at the metadata query request according to the syntax tree; and scanning the HBase database according to the start row key and the end row key to obtain a query result aiming at the metadata query request.
According to a second aspect, there is provided an apparatus for querying metadata, comprising: the response module is used for responding to the received metadata query request, analyzing the metadata query request and obtaining a plurality of partition field information associated with the metadata query request; the parsing module is used for parsing the plurality of partitioned field information to obtain a syntax tree; a determining module, configured to determine a start line key and an end line key for the metadata query request according to the syntax tree; and the scanning module is used for scanning the HBase database according to the start row key and the end row key to obtain a query result aiming at the metadata query request.
According to a third aspect, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the embodiment of the disclosure.
According to a fourth aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform a method provided by an embodiment of the disclosure.
According to a fifth aspect, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method provided by embodiments of the present disclosure.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 illustrates an exemplary system architecture suitable for a method or/and apparatus for querying metadata of embodiments of the present disclosure;
FIG. 2 is a flow diagram of a method of querying metadata, according to one embodiment of the present disclosure;
fig. 3A, fig. 3B, and fig. 3C are schematic diagrams of a first row key for a subject hierarchy, a second row key for a database hierarchy, a third row key for a table hierarchy, and a fourth row key for the partition field information in the HBase database according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a physical table, according to one embodiment of the present disclosure;
FIG. 5 is a flowchart of an execution of a method of querying metadata according to one embodiment of the present disclosure;
FIG. 6 is a block diagram of an apparatus to query metadata according to one embodiment of the present disclosure;
FIG. 7 illustrates a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Common metadata management can be divided into centralized and distributed metadata management architectures. The centralized metadata management architecture adopts a single metadata server, is simple to implement, and has the problems of single point of failure and the like. The distributed metadata management architecture disperses the metadata on a plurality of nodes, thereby solving the problems of performance bottleneck and the like of the metadata server, improving the expandability of the metadata management architecture, but having more complex realization and introducing the problem of metadata consistency.
The HBase (Hadoop Database, distributed computing Database) is a distributed storage system with high reliability, high performance, orientation and scalability, and a large-scale structured storage cluster can be built on a low-cost PC Server by utilizing the HBase technology. The storage of its metadata may not be based on the structure of the row keys and column families, but there are a series of problems.
Fig. 1 illustrates an exemplary system architecture of a method or/and apparatus for querying metadata suitable for embodiments of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present application may be applied to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application may not be used in other environments or scenarios.
As shown in fig. 1, the system architecture 100 includes a client 101, a metadata manager 102, and a data read/write manager 103.
The client 101 provides an interface for accessing the HBase and maintains a corresponding cache to speed up access of the HBase.
The metadata manager 102 stores metadata of the HBase. When the data is read or written, the client 101 obtains the metadata from the metadata manager 102, and then the client initiates a request for reading or writing the data to the corresponding data reading/writing manager according to the metadata.
The data read/write manager 103 processes the read/write request of the client 101, and is responsible for interacting with a Distributed File System (HDFS).
When data needs to be continuously written, the client 101 initiates a request for obtaining metadata to the metadata manager 102, and then the metadata manager 102 returns the address of the corresponding data read-write manager 103 to the client 101. The client 101 obtains the address returned by the metadata manager 102, then initiates a request to the corresponding data read-write manager 103 according to the address, and finally the data read-write manager 103 performs the data read-write operation and returns the data to the client 101.
It should be understood that the number of clients, metadata managers, and data read and write managers in FIG. 1 is merely illustrative. There may be any number of clients, metadata managers, and data read-write managers, as desired for implementation.
FIG. 2 is a flow diagram of a method of querying metadata, according to one embodiment of the present disclosure.
As shown in fig. 2, the method 200 may include operations S210 to S240.
In operation S210, in response to receiving a metadata query request, the metadata query request is parsed to obtain a plurality of partition field information associated with the metadata query request.
For example, the metadata manager may interact with the data read and write manager, which may interact with the distributed file system. The distributed file system may store data for the athlete, and may use the athlete's name and ability rating as the partition fields. The metadata query request may be name 'jim' and rating ≧ 97and jersey 23. Parsing the metadata query request may result in partition field information: name is jim', rating ≧ 97.
In operation S220, the partition field information is parsed to obtain a syntax tree.
According to an embodiment of the present disclosure, the syntax tree may include a logical operation relationship between partition field information.
For example, parsing multiple partition field information name ≧ jim' and rating ≧ 97 can result in the corresponding syntax tree.
In operation S230, a start line key and an end line key for the metadata query request are determined according to the syntax tree described above.
According to the syntax tree, an initial start line key and an initial end line key for a metadata query request can be determined first, and then an analyzed start line key and an analyzed end line key are obtained.
For example, according to the syntax data, an initial start row key statowkey of which the metadata query request is name 'jim' and rating ≧ 97and jersey 23 is obtained as a player and an initial end row key endrowkey is obtained as a player, and for the metadata query request, the start row key obtained after analysis may be 6player3jim297 and the end row key may be a player.
In operation S240, the HBase database is scanned according to the start line key and the end line key to obtain a query result for the metadata query request.
For example, 6 layers 3jim297 related data are obtained from the Hbase database of the beginning row key layer scanning, and the data can be used as the query result.
Through the embodiment of the disclosure, the data storage can be more compact, and the optimization scanning can be more facilitated during the query.
Fig. 3A, fig. 3B and fig. 3C are schematic diagrams of a first row key for a subject hierarchy, a second row key for a database hierarchy, a third row key for a table hierarchy, and a fourth row key for the partition field information in the HBase database according to an embodiment of the present disclosure.
As shown in fig. 3A, for each metadata, the HBase database described above may include a first row key for the topic hierarchy.
According to an embodiment of the present disclosure, the first row of keys includes the following fields arranged in sequence: module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, custom named character length, and custom naming.
For example, a first row of keys may be 4meta7default5topic6mark, where 4 is the module name character length, meta is the module name, 7 is the namespace character length, default is the namespace, 5 is the belonging level character length, topic is the belonging level, 6 is the custom named character length and mark is the custom name.
As shown in fig. 3B, for each metadata, a second row key for a database hierarchy level and a third row key for a table hierarchy level may be included in the HBase database.
According to an embodiment of the present disclosure, the second row of keys and the third row of keys each include the following fields arranged in sequence: module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, previous hierarchy custom name character length, previous hierarchy custom name, custom name character length, and custom name.
According to the embodiment of the present disclosure, the table includes a mapping table and a physical table, and the mapping table includes data source information and storage path information; the physical table includes field information.
For example, one second row of keys may be 4meta7default8database6mark 2db, where 4 is the module name character length, meta is the module name, 7 is the namespace character length, default is the namespace, 8 is the belonging level character length, database is the belonging level, 6 is the last level custom name character length, mark is the last level custom name, 2 is the custom name character length, and db is the custom name.
For example, a third row of keys may be 4meta7default5table2db6orders, where 4 is the module name character length, meta is the module name, 7 is the namespace character length, default is the namespace, 5 is the affiliated level character length, table is the affiliated level, 2 is the last level custom name character length, db is the last level custom name, 6 is the custom name character length and orders are the custom names.
As shown in fig. 3C, the field information includes non-partition field information and partition field information; wherein, the HBase database further comprises a fourth row key for the partition field information.
Implementing the fourth row of keys described above in accordance with the present disclosure includes the following fields in order: module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
For example, a fourth row key may be 9part 7default6orders10 basketballs. Wherein 9 is the module name character length, partition is the module name, 7 is the namespace character length, default is the namespace, 6 is the table name character length, orders is the table name, 10 is the partition field value character length, and basketball is the partition value.
FIG. 4 is a schematic diagram of a physical table, according to one embodiment of the present disclosure.
As shown in fig. 4, the physical table Player 400 stores therein partition field information: name, rating, and team, and non-partition field information: jersey and country.
The order of arrangement of the fourth row key rowkey in the HBase table corresponding to the physical table Player 400 in fig. 4 is a lexicographic order, which is:
9partition7default6player5henry2933B22；
9partition7default6player3jim2973A11；
9partition7default6player41ucy2953C33。
the theme of the physical table P1 player 400 is mark, and the first row of the theme is 9part 7default5topic6mark, which can be understood as the market of players in a sports tournament, where players can contract with other teams or be traded by the team to other teams. The subject may have multiple databases with the physical table Player 400 located in one of the databases db for which the second row key is 9part 7default8database6mark 2db, and the players in that database are players who may sign up with other teams. The database db may have a plurality of parallel table levels, and the third key at the level of the physical table Player 400 is 9 partial 7default5table2db6orders, which may be understood as a star Player who may sign up with other teams. At least one mapping table and at least one physical table corresponding to the mapping table can be arranged under the table hierarchy.
FIG. 5 is a flowchart of an execution of a method of querying metadata according to one embodiment of the present disclosure.
As shown in fig. 5, the execution flow 500 may include operations S501 to S508.
In operation S501, the metadata query request is parsed.
For example, the client initiates a metadata query request: selecting from Player where name ' jim ' and rating ≧ 97and jersey ' 23, the metadata manager, in response to receiving the metadata query request, parses it, gets a plurality of first partition field information: name is jim', rating ≧ 97. The Player may be the physical table Player 400 in FIG. 4.
In operation S502, the partition field is parsed. And analyzing the plurality of first partition field information to obtain a syntax tree.
For example, the syntax tree can be represented as: name jim and rating ≥ 97
In operation S503, a start row key and an end row key are determined.
For example, according to the syntax tree described above, the start line key startrowkey for the metadata query request is determined: 9partition7default6 player; and an end row key endrowkey: 9partition7default6 layers.
In one embodiment, the start row key may be optimized according to the first partition field information, for example, the optimized start row key startrowkey is: 9partition7default6player3jim 297.
In operation S504, the HBase database is scanned.
For example, the HBase database is scanned based on the start line key and the end line key to obtain the fourth line key rowkey corresponding to the physical table Player 400.
In operation S505, a fourth row of keys is determined.
For example, a fourth row of keys may be determined from the initial start row of keys: 9partition7default6player5henry2933B 22.
For example, a fourth row of keys may be determined based on the optimized start row of keys: 9partition7default6player3jim2973A 11. At this point, in effect, a fourth row of keys is skipped directly: 9partition7default6player5henry2933822, the optimized start row key can be used to reduce the scanning range.
In operation S506, the start row key is matched with the fourth row key. If the matching is successful, executing operation S507; if the matching fails, operation S508 is performed.
For example, according to a fourth row of keys: 9partition7default6player3jim2973a11, and the obtained second partition field information name is jim, rating is 97, and team is a11, and these second partition field information are matched with the first partition field information. For example, matching name jim with name jim' results in a matching result of Ture; matching the rating 97 with the rating 97, and obtaining a matching result which is also called true. Meanwhile, if the relationship between the two pieces of first partition field information is 'and', the matching of the second partition field information of the fourth row key and the first partition field information is passed, and the 9part 7default6player3jim2973a11 meeting the metadata query request is obtained, and the operation S507 is skipped.
In operation S507, a query result is obtained. Returning to operation S505.
For example, a fourth row of keys: and the 9part 7default6player3jim2973A11 is successfully matched with the start row key, and the metadata corresponding to the row key is used as a query result. And jumps back to operation S505.
For example, after the matching operation for one fourth row key 9 partial 7default6 layer3jim2973a11 is finished, the matching operation is performed for the following fourth row key 9 partial 7default6 layer4 luma 2953C 33. According to the fourth row of keys: 9partition7default6player4 size 2953C33, and the obtained second partition field information name is size, rating is 95, and team is C33, and these second partition field information are matched with the first partition field information. For example, matching name lucy with name 'jim' yields a match result of False; matching the rating of 95 with the rating of more than or equal to 97 to obtain a matching result which is also False. Meanwhile, if the relationship between the two pieces of first partition field information is 'and', the matching between the second partition field information of the fourth row key and the first partition field information is failed, and the result that the 9partition7default6 layer4lucy2953C33 does not conform to the metadata query request is obtained, and the operation S508 is skipped.
In operation S508, the end row key is matched with the fourth row key. If the matching is successful, the process is ended, and if the matching is failed, the operation returns to operation S505 to determine another fourth row key.
For example, the fourth row key 9partition7default6player4 size 2953C33 failed to match the start row key and matched the end row key, resulting in a matching failure. Returning to operation S505, a fourth row of keys is determined.
Through the embodiment of the disclosure, the corresponding physical table can be filtered, and the accurate query result can be returned as far as possible.
FIG. 6 is a block diagram of an apparatus to query metadata according to one embodiment of the present disclosure.
As shown in fig. 6, the apparatus for querying metadata includes a response module 610, a parsing module 620, a determination module 630 and a scanning module 640.
The response module 610 is configured to, in response to receiving the metadata query request, parse the metadata query request to obtain a plurality of partition field information associated with the metadata query request.
And the parsing module 620 is configured to parse the plurality of partition field information to obtain a syntax tree.
The determining module 630 is configured to determine a start line key and an end line key for the metadata query request according to the syntax tree.
And the scanning module 640 is configured to scan the HBase database according to the start line key and the end line key to obtain a query result for the metadata query request.
As an alternative embodiment, for each metadata, the HBase database includes a first row key for the subject hierarchy, a second row key for the database hierarchy, and a third row key for the table hierarchy; the table comprises a mapping table and a physical table, wherein the mapping table comprises data source information and storage path information; the physical table includes field information.
As an alternative embodiment, the first row of keys includes the following fields arranged in sequence: module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, custom named character length, and custom naming.
As an alternative embodiment, the second row of keys and the third row of keys each include the following fields arranged in sequence: module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, previous hierarchy custom name character length, previous hierarchy custom name, custom name character length, and custom name.
As an optional embodiment, the field information includes non-partition field information and partition field information; wherein, the HBase database further includes a fourth row of keys for the partition field information, and the fourth row of keys sequentially includes the following fields: module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
As an alternative embodiment, the scanning module includes: and the matching unit is used for respectively matching the start row key and the end row key with a fourth row key in the HBase database to obtain a query result aiming at the metadata query request.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
Fig. 7 illustrates a schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 7, the device 700 comprises a computing unit 701, which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM)702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the disclosure, the data storage can be more compact, and the optimization scanning can be more facilitated during the query.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (15)
1. A method of querying metadata, comprising:
in response to receiving a metadata query request, parsing the metadata query request to obtain a plurality of partition field information associated with the metadata query request;
analyzing the plurality of partition field information to obtain a syntax tree;
determining a start line key and an end line key for the metadata query request according to the syntax tree; and
and scanning the HBase database according to the start row key and the end row key to obtain a query result aiming at the metadata query request.
2. The method of claim 1, wherein, for each metadata, the HBase database includes a first row key for a subject hierarchy, a second row key for a database hierarchy, and a third row key for a table hierarchy;
the table comprises a mapping table and a physical table, wherein the mapping table comprises data source information and storage path information; the physical table includes field information.
3. The method of claim 2, wherein the first row of keys comprises the following fields arranged in order:
module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, custom named character length, and custom naming.
4. The method of claim 2, wherein the second and third rows of keys each comprise the following fields arranged in order:
module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, previous hierarchy custom name character length, previous hierarchy custom name, custom name character length, and custom name.
5. The method of claim 2, wherein the field information comprises non-partition field information and partition field information;
wherein the HBase database further comprises a fourth row of keys for the partition field information, and the fourth row of keys sequentially comprises the following fields:
module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
6. The method according to claim 5, wherein the scanning the HBase database according to the start row key and the end row key to obtain the query result for the metadata query request comprises:
and respectively matching the starting row key and the ending row key with a fourth row key in the HBase database to obtain a query result aiming at the metadata query request.
7. An apparatus for querying metadata, comprising:
the response module is used for responding to the received metadata query request, analyzing the metadata query request and obtaining a plurality of partition field information associated with the metadata query request;
the analysis module is used for analyzing the plurality of partition field information to obtain a syntax tree;
the determining module is used for determining a start line key and an end line key aiming at the metadata query request according to the syntax tree;
and the scanning module is used for scanning the HBase database according to the start row key and the end row key to obtain a query result aiming at the metadata query request.
8. The apparatus of claim 7, wherein, for each metadata, the HBase database comprises a first row key for a subject hierarchy, a second row key for a database hierarchy, and a third row key for a table hierarchy;
the table comprises a mapping table and a physical table, wherein the mapping table comprises data source information and storage path information; the physical table includes field information.
9. The apparatus of claim 8, wherein the first row of keys comprises the following fields arranged in order:
module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, custom named character length, and custom naming.
10. The apparatus of claim 8, wherein the second and third rows of keys each comprise the following fields arranged in order:
module name character length, module name, namespace character length, namespace, affiliated hierarchy character length, affiliated hierarchy, previous hierarchy custom name character length, previous hierarchy custom name, custom name character length, and custom name.
11. The apparatus of claim 8, wherein the field information comprises non-partition field information and partition field information;
wherein the HBase database further comprises a fourth row of keys for the partition field information, and the fourth row of keys sequentially comprises the following fields:
module name character length, module name, namespace character length, namespace, table name character length, table name, partition field value character length, and partition value.
12. The apparatus of claim 11, wherein the scanning module comprises:
and the matching unit is used for respectively matching the start row key and the end row key with a fourth row key in the HBase database to obtain a query result aiming at the metadata query request.
13. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-6.
15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110645140.4A CN113297273B (en) | 2021-06-09 | 2021-06-09 | Method and device for inquiring metadata and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110645140.4A CN113297273B (en) | 2021-06-09 | 2021-06-09 | Method and device for inquiring metadata and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113297273A true CN113297273A (en) | 2021-08-24 |
| CN113297273B CN113297273B (en) | 2024-03-01 |
Family
ID=77327757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110645140.4A Active CN113297273B (en) | 2021-06-09 | 2021-06-09 | Method and device for inquiring metadata and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297273B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104252536A (en) * | 2014-09-16 | 2014-12-31 | 福建新大陆软件工程有限公司 | Hbase-based internet log data inquiring method and device |
| CN104778406A (en) * | 2015-04-08 | 2015-07-15 | 公安部第一研究所 | Method for uniformly naming malicious codes based on file fingerprint and system thereof |
| US20150317345A1 (en) * | 2012-11-27 | 2015-11-05 | Nokia Solutions And Networks Oy | Multiple fields parallel query method and corresponding storage organization |
| CN106326361A (en) * | 2016-08-10 | 2017-01-11 | 中国农业银行股份有限公司 | HBase database-based data inquiry method and device |
| CN106326429A (en) * | 2016-08-25 | 2017-01-11 | 武汉光谷信息技术股份有限公司 | Hbase second-level query scheme based on solr |
| CN106682139A (en) * | 2016-12-19 | 2017-05-17 | 深圳盒子支付信息技术有限公司 | Method and system for achieving HBase multi-condition query based on Solr |
| CN109299102A (en) * | 2018-10-23 | 2019-02-01 | 中国电子科技集团公司第二十八研究所 | A kind of HBase secondary index system and method based on Elastcisearch |
| CN109299110A (en) * | 2018-11-09 | 2019-02-01 | 东软集团股份有限公司 | Data query method, apparatus, storage medium and electronic equipment |
| CN109785034A (en) * | 2018-11-13 | 2019-05-21 | 北京码牛科技有限公司 | User's portrait generation method, device, electronic equipment and computer-readable medium |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110347722A (en) * | 2019-07-11 | 2019-10-18 | 软通智慧科技有限公司 | Data capture method, device, equipment and storage medium based on HBase |
| CN110362549A (en) * | 2019-06-17 | 2019-10-22 | 平安普惠企业管理有限公司 | Log memory search method, electronic device and computer equipment |
| CN110427366A (en) * | 2019-06-29 | 2019-11-08 | 苏州浪潮智能科技有限公司 | Index synchronous method, apparatus and system based on the phoenix transformation old system of hbase |
| US20200057781A1 (en) * | 2018-08-20 | 2020-02-20 | Salesforce.org | Mapping and query service between object oriented programming objects and deep key-value data stores |
| CN111552753A (en) * | 2020-04-24 | 2020-08-18 | 中国科学院空天信息创新研究院 | Global massive remote sensing data organization and management method and system |
| CN112860695A (en) * | 2021-02-08 | 2021-05-28 | 北京百度网讯科技有限公司 | Monitoring data query method, device, equipment, storage medium and program product |
-
2021
- 2021-06-09 CN CN202110645140.4A patent/CN113297273B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150317345A1 (en) * | 2012-11-27 | 2015-11-05 | Nokia Solutions And Networks Oy | Multiple fields parallel query method and corresponding storage organization |
| CN104252536A (en) * | 2014-09-16 | 2014-12-31 | 福建新大陆软件工程有限公司 | Hbase-based internet log data inquiring method and device |
| CN104778406A (en) * | 2015-04-08 | 2015-07-15 | 公安部第一研究所 | Method for uniformly naming malicious codes based on file fingerprint and system thereof |
| CN106326361A (en) * | 2016-08-10 | 2017-01-11 | 中国农业银行股份有限公司 | HBase database-based data inquiry method and device |
| CN106326429A (en) * | 2016-08-25 | 2017-01-11 | 武汉光谷信息技术股份有限公司 | Hbase second-level query scheme based on solr |
| CN106682139A (en) * | 2016-12-19 | 2017-05-17 | 深圳盒子支付信息技术有限公司 | Method and system for achieving HBase multi-condition query based on Solr |
| US20200057781A1 (en) * | 2018-08-20 | 2020-02-20 | Salesforce.org | Mapping and query service between object oriented programming objects and deep key-value data stores |
| CN109299102A (en) * | 2018-10-23 | 2019-02-01 | 中国电子科技集团公司第二十八研究所 | A kind of HBase secondary index system and method based on Elastcisearch |
| CN109299110A (en) * | 2018-11-09 | 2019-02-01 | 东软集团股份有限公司 | Data query method, apparatus, storage medium and electronic equipment |
| CN109785034A (en) * | 2018-11-13 | 2019-05-21 | 北京码牛科技有限公司 | User's portrait generation method, device, electronic equipment and computer-readable medium |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110362549A (en) * | 2019-06-17 | 2019-10-22 | 平安普惠企业管理有限公司 | Log memory search method, electronic device and computer equipment |
| CN110427366A (en) * | 2019-06-29 | 2019-11-08 | 苏州浪潮智能科技有限公司 | Index synchronous method, apparatus and system based on the phoenix transformation old system of hbase |
| CN110347722A (en) * | 2019-07-11 | 2019-10-18 | 软通智慧科技有限公司 | Data capture method, device, equipment and storage medium based on HBase |
| CN111552753A (en) * | 2020-04-24 | 2020-08-18 | 中国科学院空天信息创新研究院 | Global massive remote sensing data organization and management method and system |
| CN112860695A (en) * | 2021-02-08 | 2021-05-28 | 北京百度网讯科技有限公司 | Monitoring data query method, device, equipment, storage medium and program product |
Non-Patent Citations (2)
| Title |
|---|
| 崔晨;郑林江;韩凤萍;何牧君;: "基于内存的HBase二级索引设计", 计算机应用, no. 06, pages 64 - 70 * |
| 陈希林;马丁;: "针对微博信息分析的HBase存储结构设计", 信息网络安全, no. 09, pages 275 - 279 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113297273B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7269913B2 (en) | Knowledge graph construction method, device, electronic device, storage medium and computer program | |
| US20150026189A1 (en) | Index structure for a relational database table | |
| JP2022046759A (en) | Retrieval method, device, electronic apparatus and storage medium | |
| JP7300475B2 (en) | Entity Relationship Mining Method, Apparatus, Electronic Device, Computer Readable Storage Medium and Computer Program | |
| JP2021535473A (en) | Token matching in a large document corpus | |
| CN113760839A (en) | Log data compression processing method and device, electronic equipment and storage medium | |
| US11397750B1 (en) | Automated conflict resolution and synchronization of objects | |
| CN114816393A (en) | Information generation method, device, equipment and storage medium | |
| CN111290714B (en) | Data reading method and device | |
| CN111435406A (en) | Method and device for correcting database statement spelling errors | |
| CN111666302A (en) | User ranking query method, device, equipment and storage medium | |
| CN113297273B (en) | Method and device for inquiring metadata and electronic equipment | |
| US9367889B2 (en) | System and method for propagating scene information to renderers in a multi-user, multi-scene environment | |
| KR102531507B1 (en) | Method, device, equipment and storage medium for outputting information | |
| US11966401B2 (en) | Query tree labeling and processing | |
| CN115328917A (en) | Query method, device, equipment and storage medium | |
| CN115080607A (en) | Method, device, equipment and storage medium for optimizing structured query statement | |
| CN114238335A (en) | Buried point data generation method and related equipment thereof | |
| US11256679B2 (en) | Systems and methods for storing object state on hash chains | |
| US20230214394A1 (en) | Data search method and apparatus, electronic device and storage medium | |
| US9477699B2 (en) | Static row identifier space partitioning for concurrent data insertion in delta memory store | |
| CN116383412B (en) | Functional point amplification method and system based on knowledge graph | |
| JP7293544B2 (en) | Q&A system update processing method and device | |
| CN113642295B (en) | Page typesetting method, device and computer program product | |
| EP4131017A2 (en) | Distributed data storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |