WO2026001091A1 - 一种数据查询方法与设备 - Google Patents

一种数据查询方法与设备

Info

Publication number
WO2026001091A1
WO2026001091A1 PCT/CN2025/082684 CN2025082684W WO2026001091A1 WO 2026001091 A1 WO2026001091 A1 WO 2026001091A1 CN 2025082684 W CN2025082684 W CN 2025082684W WO 2026001091 A1 WO2026001091 A1 WO 2026001091A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
query
node
information
data block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/082684
Other languages
English (en)
French (fr)
Inventor
陈文辉
凌民光
刘怀忠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2026001091A1 publication Critical patent/WO2026001091A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution

Definitions

  • This application relates to the field of big data technology, and in particular to a data query method and device.
  • Point queries refer to querying data based on a single indexed field
  • range queries refer to querying data based on a small number of indexed fields (e.g., less than or equal to 3)
  • multidimensional queries refer to querying data based on multiple indexed fields (e.g., greater than 3).
  • point queries require millisecond-level latency
  • range queries require sub-second (less than 1 second) latency
  • multidimensional queries require second-level latency.
  • This application provides a data query method and device to reduce resource consumption.
  • a data query method is provided, which can be executed by a second node, also known as a data query device, to perform data query operations.
  • the second node can query required data from a database, which includes multiple data tables used for data analysis.
  • the data analysis operations involve retrieving and analyzing data from the database.
  • the second node receives a first request to retrieve data from the database, including a query statement.
  • the second node determines information about at least one data block.
  • This at least one data block is a data block within the data blocks corresponding to the first data table that meets the data query conditions.
  • the first data table and the data query conditions are determined based on the query statement, and the description information indicates the data blocks corresponding to the first data table.
  • the second node reads the at least one data block from the database.
  • the second node can determine information about at least one data block that meets the data query conditions from the data blocks corresponding to the first data table based on the description information of the first data table.
  • This at least one data block is a data block that meets the data query conditions; that is, at least one data block includes the data required by the first request. Therefore, reading at least one data block can obtain the data requested by the first request.
  • this embodiment implements data query services based on the data tables used for data analysis. Data analysis typically queries data based on a large number of dimensions of query conditions and performs data analysis on the queried data; therefore, data analysis can also be called multidimensional data analysis or multidimensional query services.
  • Data query services are used to obtain data from the database.
  • Data query services typically perform data queries based on a small number of dimensions of query conditions; for example, data query services can include point queries, range queries, or multidimensional queries.
  • data query services can include point queries, range queries, or multidimensional queries.
  • the embodiments of this application enable data tables originally used for data analysis to be used for scenarios such as point queries, range queries, or multidimensional queries.
  • This allows multiple data query scenarios to share the same data table using a single big data query technology. Therefore, different data query scenarios do not need to use their own specific big data query technologies, and thus resources do not need to be deployed separately for each technology. Instead, resources can be deployed for a single, shared big data query technology, reducing resource overhead.
  • the description information of the first data table indicates the corresponding data blocks, such as the data included in the data block. Therefore, the second node can combine this description information with the data query conditions to determine the data blocks in the first data table that meet the data query conditions. This reduces the complexity of the data query process and
  • the second node can adaptively determine the at least one data block that meets the data query conditions based on whether or not it stores descriptive information. If the second node stores descriptive information, it can determine the information of at least one data block based on the stored descriptive information. This allows the second node to determine the at least one data block that meets the data query conditions locally, saving the communication overhead of requesting other nodes to determine the at least one data block that meets the data query conditions, thus improving data query efficiency. Alternatively, if the second node does not store descriptive information, it can send a second request to the first node and receive a first response from the first node.
  • the first node stores descriptive information of data tables in the database and determines the information of at least one data block that meets the data query conditions based on the descriptive information.
  • the second request includes the identifier of the first data table and the data query conditions, and the first response includes the information of at least one data block.
  • the information for at least one data block includes one or more of the following: the identifier of the table file to which each data block belongs, the address offset of each data block within its respective table file, or the length of each data block. This information allows the determination of the storage location of at least one data block in the database, thereby enabling successful retrieval of at least one data block from the database.
  • the second node when the first data table is of type 1, the second node obtains and stores descriptive information from the first node. Since the data volume corresponding to the data table of type 1 is less than or equal to a first threshold, the time required to determine the data block that meets the data query conditions from the data block corresponding to the data table of type 1 is relatively short, and the impact on other functions in the second node is also minimal. After storing the descriptive information of the data table of type 1, the second node can determine the data block that meets the data query conditions locally, saving the communication overhead of requesting other nodes to determine the data query conditions and helping to improve data query efficiency.
  • the second node may receive information about a first duration from the first node, and determine that the type of the first data table is a first type when the first duration is less than or equal to a second threshold.
  • the first duration is the time consumed by the first node to determine at least one data block. If the first duration is less than or equal to the second threshold, it means that the time required to determine whether the data query conditions are met from the data blocks corresponding to the data table of the first type is relatively short. This reflects that the amount of data corresponding to the first data table is relatively small, and thus the type of the first data table can be determined to be the first type.
  • the second threshold is determined based on the first round-trip time (RTT).
  • the first RTT is the duration between the sending time of the second request and the receiving time of the first response. Determining the second threshold based on the first RTT allows for a measurement of the relationship between the first duration and the first RTT. For example, when the first duration is closer to the second RTT, it indicates that most of the time in the first RTT is spent determining data blocks that meet the data query conditions, with data transmission between the second node and the first node occupying only a small portion of the time. In this case, requesting the first node to perform the determination process can save more resources for the second node.
  • the first duration is closer to the second RTT, it indicates that most of the time in the first RTT is spent on data transmission between the second node and the first node. In this case, requesting the first node to perform the determination process can save less resources for the second node. Therefore, by setting the second threshold based on the first RTT, when the second node determines whether to classify the first data table as a first type of data table, the amount of resource consumption saved by the second node is essentially taken into consideration, which helps to balance the resource consumption and communication overhead of the second node.
  • the descriptive information includes one or more of the following: Metadata, used to indicate a first index field included in the first data table, wherein the first data table includes multiple partitions, the multiple partitions being determined based on the value of the first index field; First indication information, used to indicate the table files corresponding to the multiple partitions, wherein the table file corresponding to the first partition among the multiple partitions includes the data corresponding to the first partition, and one of the table files corresponding to the multiple partitions includes one or more data blocks, wherein one of the data blocks includes the value of one or more index fields; Second indication information, used to indicate the value range of the index field of the data block included in one of the table files corresponding to the multiple partitions. With this information, the second node can accurately determine at least one data block from the data blocks corresponding to the first data table that satisfies the data query conditions.
  • the data query conditions include the value of at least one index field.
  • the second node determines, based on metadata, that at least one index field includes the first index field, it can determine a second partition from multiple partitions.
  • the second partition is the partition where the value of the first index field (at least one index field) is located.
  • the second node can determine the table file corresponding to the second partition based on first indication information, and finally, based on second indication information, determine information about at least one data block, where the at least one data block includes the value of one or more of the at least one index field.
  • the second node can filter out partitions that do not meet the data query conditions based on the first index field used to partition the partitions, reducing the number of partitions that need to be queried, thereby improving data query efficiency.
  • the data query conditions include the value of at least one index field.
  • the second node determines, based on metadata, that at least one index field does not include the first index field, it determines the table file corresponding to the first data table based on the first indication information. Finally, based on the second indication information, it determines the information of at least one data block, where the at least one data block includes the value of one or more of the at least one index field.
  • the second node determines that the index field in the data query conditions does not include the first index field, it queries the data blocks of all table files corresponding to the first data table to identify at least one data block that satisfies the data query conditions. This filters out data blocks in the first data table that do not meet the data query conditions, reducing the number of data blocks that need to be read and improving data query performance.
  • embodiments of this application also provide a data query method, which can be executed by a first node.
  • the first node receives a second request from a second node, the second request including an identifier of a first data table and data query conditions.
  • the first data table includes multiple data tables in a database, which are used for data analysis operations.
  • the data analysis operations are used to retrieve data from the database and analyze the retrieved data.
  • the first node determines at least one data block in the data blocks corresponding to the first data table that meets the data query conditions.
  • the description information is used to indicate the data blocks corresponding to the first data table.
  • the first node sends a first response to the second node, the first response including first indication information, the first indication information being used to indicate at least one data block.
  • the first node can determine the information of at least one data block that meets the data query conditions from the data blocks corresponding to the first data table based on the description information of the first data table.
  • this embodiment implements data query services based on the data tables used for data analysis. Therefore, this embodiment allows data tables originally used for data analysis to be used for point queries, range queries, or multi-dimensional queries, enabling multiple data query scenarios to share a single database technology data table. This eliminates the need for separate deployment of storage resources for storing data tables corresponding to different database technologies, as required in related technologies, thus reducing the resource consumption required for data table storage.
  • the description information of the first data table can indicate the data blocks corresponding to the first data table, such as indicating the data included in the data block. Therefore, the data blocks in the first data table that meet the data query conditions can be determined by combining the description information and the data query conditions, resulting in low complexity of the data query process and improved data query performance.
  • the first node stores descriptive information of the data tables in the database.
  • the first node can obtain this descriptive information in several ways.
  • One way is that, in response to a preload instruction, the first node reads the table files corresponding to multiple data tables in the database and determines the descriptive information of these tables based on these files. In this way, the first node can preload the descriptive information of all data tables.
  • Another way is that, based on a third request from a third node, the first node reads the table file corresponding to a second data table and determines the descriptive information of this second data table based on this file.
  • the third request instructs the third node to update the second data table.
  • the third node maintains the database. In this way, the first node can update the descriptive information promptly when data in the database is updated, improving the accuracy of the descriptive information and reducing the probability of data query failures.
  • the information of at least one data block includes one or more of the following: the identifier of the table file to which each data block belongs; the address offset of each data block in the table file to which it belongs; or, the length of each data block.
  • information of a first duration is sent to the second node, the first duration being the duration consumed by the first node in determining at least one data block.
  • the descriptive information includes one or more of the following: Metadata, used to indicate a first index field included in the first data table, wherein the first data table includes multiple partitions, the multiple partitions being determined based on the value of the first index field; First indication information, used to indicate the table files corresponding to the multiple partitions respectively, wherein the table file corresponding to the first partition among the multiple partitions includes the data corresponding to the first partition, and one of the table files corresponding to the multiple partitions includes one or more data blocks, wherein one of the data blocks includes the value of one or more index fields; Second indication information, used to indicate the value range of the index field of the data block included in one of the table files corresponding to the multiple partitions.
  • the data query conditions include the value of at least one index field.
  • the first node determines that at least one index field includes a first index field based on metadata
  • it can determine a second partition from multiple partitions.
  • the second partition is the partition where the value of the first index field in at least one index field is located.
  • Based on first indication information it determines the table file corresponding to the second partition.
  • Based on second indication information it determines the information of at least one data block, where the at least one data block includes the value of one or more index fields in at least one index field.
  • the data query conditions include the value of at least one index field.
  • the first node determines, based on metadata, that at least one index field does not include the first index field, it can determine the table file corresponding to the first data table based on the first indication information, and finally determine the information of at least one data block based on the second indication information.
  • the at least one data block includes the value of one or more of the at least one index field.
  • embodiments of this application also provide a data query device, which has the functions of the method embodiment of the first aspect described above.
  • the beneficial effects can be found in the description of the first aspect, and will not be repeated here.
  • the functions can be implemented by hardware or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the data query device includes a receiving module, a determining module, and a data reading module. These modules can perform the corresponding functions in the method examples of the first aspect described above, as detailed in the method examples, and will not be repeated here.
  • the receiving module receives a first request to retrieve data from a database, including a query statement.
  • the database includes multiple tables used for data analysis, which retrieve and analyze data.
  • the determining module identifies at least one data block based on the description information of the first data table. This at least one data block is a data block within the corresponding data table that meets the data query conditions.
  • the first data table and the query conditions are determined based on the query statement, and the description information indicates the data block corresponding to the first data table.
  • the data reading module reads at least one data block from the database based on the information of the at least one data block.
  • information about at least one data block that meets the data query conditions can be determined from the data blocks corresponding to the first data table based on the description information of the first data table.
  • This at least one data block is a data block that meets the data query conditions; that is, at least one data block includes the data required by the first request. Therefore, reading at least one data block can obtain the data requested by the first request.
  • this embodiment of the application implements a data query service based on the data tables used for data analysis. Data analysis typically queries data based on a large number of dimensions of query conditions and performs data analysis on the queried data; therefore, data analysis can also be called multidimensional data analysis or multidimensional query service.
  • Data query service is used to obtain data from the database.
  • Data query service typically performs data queries based on a small number of dimensions of query conditions; for example, data query service can be a point query, a range query, or a multidimensional query service, etc.
  • data query service can be a point query, a range query, or a multidimensional query service, etc.
  • the embodiments of this application enable data tables originally used for data analysis to be used for scenarios such as point queries, range queries, or multidimensional queries. This allows multiple data query scenarios to share a single database technology data table, eliminating the need for separate storage resources to store data tables corresponding to different database technologies, as required in related technologies. Therefore, the embodiments of this application reduce the resource consumption required for data table storage.
  • the description information of the first data table can indicate the data blocks corresponding to the first data table, such as the data included in the data block. Therefore, by combining this description information with the data query conditions, the data blocks in the first data table that meet the data query conditions can be determined, resulting in lower complexity in the data query process and improved data query performance.
  • this application also provides a data query device that implements the functions described in the method embodiment of the second aspect above.
  • the beneficial effects are described in the second aspect and will not be repeated here.
  • the functions can be implemented by hardware or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the data query device includes a receiving module, a determining module, and a sending module. These modules can perform the corresponding functions described in the method example of the second aspect above, as detailed in the method example, and will not be repeated here.
  • the receiving module is used to receive a second request from a second node, the second request including the identifier of a first data table and data query conditions; wherein, the first data table is included in a database, and the database in the database system includes multiple data tables, which are used for data analysis operations, and the data analysis operations are used to retrieve data from the database and analyze the retrieved data.
  • the determining module is used to determine at least one data block in the data blocks corresponding to the first data table that meets the data query conditions, based on the stored description information of the first data table, the description information being used to indicate the data block corresponding to the first data table.
  • the sending module is used to send a first response to the second node, the first response including first indication information, the first indication information being used to indicate at least one data block.
  • the data query device can determine information of at least one data block that meets the data query conditions from the data blocks corresponding to the first data table based on the description information of the first data table.
  • this embodiment implements data query functionality based on the data tables used for data analysis. Therefore, this embodiment allows data tables originally used for data analysis to be used for point queries, range queries, or multidimensional queries, enabling multiple data query scenarios to share a single database technology data table. This eliminates the need for separate storage resources for storing data tables corresponding to different database technologies, as required in related technologies, thus reducing the resource consumption required for data table storage.
  • the description information of the first data table can indicate the data blocks corresponding to the first data table, such as the data included in the data block. Therefore, the data blocks in the first data table that meet the data query conditions can be determined by combining the description information and the data query conditions. This reduces the complexity of the data query process and improves data query performance.
  • this application also provides a big data processing system, including the data query devices described in the third and fourth aspects above.
  • the big data processing system further includes a data analysis node for performing data analysis tasks and maintaining the database.
  • this application also provides a computer device, which includes a processor and a memory, and may further include a communication interface.
  • the processor executes program instructions in the memory to perform the method provided by the first aspect or any possible implementation thereof.
  • the processor executes program instructions in the memory to perform the method provided by the second aspect or any possible implementation thereof.
  • the memory is coupled to the processor and stores computer program instructions and data necessary for data querying.
  • the communication interface is used to communicate with other devices, such as receiving a first request, a second request, a first response, or sending a second request, a first response, etc.
  • this application also provides a computer device, which includes an acceleration device and a processor.
  • it may also include a memory and a communication interface.
  • the processor and the acceleration device cooperate to execute the method provided in the first aspect or any possible implementation thereof.
  • the acceleration device executes the method provided in the first aspect or any possible implementation thereof.
  • the memory is coupled to the processor and stores computer program instructions and data required during data querying.
  • the communication interface is used to communicate with other devices, such as receiving a first request, a second request, a first response, etc.
  • the processor and the acceleration device cooperate to execute the method provided in the second aspect or any possible implementation thereof.
  • the acceleration device executes the method provided in the second aspect or any possible implementation thereof.
  • the communication interface is used to communicate with other devices, such as sending a second request, a first response, etc.
  • this application provides a computer system comprising at least one computer device.
  • Each computer device includes a memory and a processor.
  • the processor of at least one computer device is configured to access code in the memory to execute the method provided by the first aspect or any possible implementation thereof, or the processor of at least one computer device is configured to access code in the memory to execute the method provided by the second aspect or any possible implementation thereof.
  • this application provides a computer-readable storage medium that, when executed by a computer device, allows the computer device to perform the method provided in the first aspect or any possible implementation thereof, or to perform the method provided in the second aspect or any possible implementation thereof.
  • the storage medium stores computer program instructions.
  • the storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).
  • this application provides a computer program product comprising computer program instructions.
  • the computer device When executed by a computer device, the computer device performs the method provided in the first aspect or any possible implementation thereof, or performs the method provided in the second aspect or any possible implementation thereof.
  • the computer program product may be a software installation package.
  • the computer program product may be downloaded and executed on a computing device.
  • this application also provides a computer chip or chip system connected to a memory, which is used to read and execute computer program instructions stored in the memory, to execute the methods in the first aspect and various possible implementations of the first aspect, or to execute the methods in the second aspect and various possible implementations of the second aspect.
  • beneficial effects of the third to eleventh aspects and their implementation methods can be referred to the description of the beneficial effects of the first or second aspect and any of its implementation methods.
  • Figure 1 is a schematic diagram of an application scenario of the solution provided in the embodiment of this application.
  • FIGS. 2A and 2B are schematic diagrams of the data query process in related technologies
  • Figure 3 is a schematic diagram of the system architecture of the database system provided in the application embodiment
  • Figure 4 is a flowchart illustrating a data query method provided in an embodiment of this application.
  • FIG. 5 is a schematic diagram of another data query method provided in the embodiments of this application.
  • Figure 6 is a schematic diagram of the process of obtaining description information at the first node according to an embodiment of this application.
  • FIG. 7 is a schematic flowchart of another data query method provided in the embodiments of this application.
  • Figure 8 is a flowchart illustrating the process of storing description information of the first data table in the second node according to an embodiment of this application;
  • Figure 9 is a flowchart illustrating the process of determining information of at least one data block according to an embodiment of this application.
  • Figure 10 is a schematic diagram of a process for determining a data block according to an embodiment of this application.
  • Figure 11 is a schematic diagram of a data query device provided in an embodiment of this application.
  • Figure 12 is a schematic diagram of another data query device provided in an embodiment of this application.
  • Figure 13 is a schematic diagram of another structure of the data query device provided in the embodiment of this application.
  • the number of nouns refers to "singular nouns or plural nouns," that is, "one or more.”
  • “At least one” means one or more, and “more than one” means two or more.
  • “And/or” describes the relationship between related objects, indicating that three relationships can exist.
  • a and/or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural.
  • the character “/” generally indicates that the related objects before and after are in an "or” relationship.
  • A/B means: A or B.
  • “At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.
  • At least one of a, b, or c means: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.
  • first and second are used to distinguish multiple objects and are not used to limit the size, content, order, timing, priority, or importance of the multiple objects.
  • first data block and the third data block can be the same data block or different data blocks.
  • the numbering of steps in the various embodiments described in this application is only to distinguish different steps and is not used to limit the order of steps.
  • a database is a repository for organizing, storing, and managing data according to a specific data structure. It stores data in a specific format, allowing users to perform CRUD operations (creating, deleting, modifying, and querying) on the stored data.
  • a database is a collection of files; it can essentially be understood as a file system.
  • a database refers to the data warehouse within a big data platform, used to store the data involved in the big data platform.
  • a data table is an object used to store data in a database.
  • a data table can also be understood as a collection of structured data, a form of structured data organization, typically stored in rows and columns. Each row represents a record in the table, and each column represents the value of an indexed field or attribute within that record.
  • a user information table might include indexed fields such as: user identity (ID), name, age, or address, and each row would contain specific information about a single user.
  • Table files are physical files used in a database to store data from tables; they can also be called data files.
  • files typically refer to physical files in the file system, used to store various data structures and content, such as table data, indexes, and logs.
  • Each database usually consists of multiple files, each responsible for storing data of a specific type or table.
  • files may include data files (storing table data), log files (recording database operation logs), and index files (storing database index information).
  • each table is usually stored as a file or a set of related files, containing the actual data of all records in the table. Data is read and written through table files in the database system to achieve persistent data storage and efficient access.
  • the physical storage format and specific implementation of table files depend on the specific business scenario; different business scenarios may have different file organization structures and storage strategies.
  • HBase a key-value query and storage engine
  • the data tables are HBase format tables
  • the table files are HFile files.
  • the combination of HBase and HFile formats enables fast point-to-point and range queries.
  • Spark a multidimensional query and storage engine
  • the data tables are in Parquet format
  • the table files are in Parquet format.
  • the combination of Spark and Parquet format can achieve efficient multidimensional data analysis.
  • a data block is a segment of data within a table file.
  • a data block can be a row group within a Parquet file.
  • a query statement is a command or instruction used to retrieve data from a database. It specifies which data to return and how to filter, sort, and combine it.
  • a common query command is the SELECT command (used to select data to retrieve from the database).
  • query commands include clauses, such as FROM and WHERE, which are two key clauses for constructing a query statement. FROM specifies the identifier of the table or view to query, and WHERE filters data rows that meet specific conditions.
  • index field also known as a field or attribute of data in a table, is one or more columns in a table. Index fields are typically used to speed up query operations by creating an index structure.
  • a user information table might include index fields such as user ID, name, age, or address.
  • An index structure can be understood as a data structure, similar to a table of contents in a book, that allows for quick location of specific content. Common index data structures include B-trees, hash tables, and full-text indexes, each suitable for different query patterns and data characteristics.
  • Multidimensional data analysis also known as online analytical processing (OLAP) is a technique and methodology for exploring and analyzing complex data. Its core concept is to analyze data from multiple perspectives to gain a deeper understanding of relationships, trends, and patterns.
  • a multidimensional data model is the foundation of multidimensional data analysis, consisting of multiple dimensions and measures. Dimensions are descriptive attributes of data; an indexed field can serve as a dimension. Measures are the numerical data to be analyzed. Dimensions are the primary basis for data analysis and are typically descriptive attributes or categories, such as time, geographic location, or product category. Different combinations of dimensions form a multidimensional data space. Measures are the numerical data analyzed in multidimensional data analysis, such as sales revenue, quantity, and profit. In multidimensional data analysis, data is often organized into a cubic form, also known as a data cube or OLAP cube.
  • the data query process includes query plan generation and query execution.
  • a query plan is a detailed set of steps and strategies generated during the execution of an SQL query, describing how to retrieve the required data. It typically includes access paths, join methods, the order in which filter conditions are applied, and sorting methods.
  • Data skipping is a technique for optimizing query plans. When performing data skipping, files that do not meet the data query conditions can be filtered out based on the values of index fields. This optimization process is also known as "pruning," which can be further divided into partition pruning and data pruning. Partition pruning removes partitions of a data table that do not meet the query conditions, while data pruning removes files within a partition that do not meet the query conditions. In this embodiment, data skipping can directly filter out data blocks that do not meet the data query conditions, providing finer data granularity compared to related technologies and effectively improving the speed of data queries.
  • FIG. 1 illustrates a scenario in which the method of this application embodiment can be applied.
  • This scenario may include a server 101 and a terminal device 102.
  • the server 101 can be a server of a big data platform, which may include a computing cluster composed of one or more computer devices 1011. Users can send corresponding business requests to the server 101 through the terminal device 102 to use the services provided by the big data platform, such as data query business requests or data analysis business requests.
  • a big data platform also known as a big data processing system or big data system, is capable of effectively processing, storing, and analyzing massive amounts of data.
  • the core of a big data platform is the big data technology stack, which refers to a collection of technologies and tools used to process and analyze large-scale data. These tools can effectively collect, store, process, and analyze large amounts of data to extract valuable information. See Figure 1; the following are some common key components of the big data technology stack:
  • Data collection and preprocessing services This stage involves collecting data from various sources and performing necessary cleaning and transformation to ensure the quality and applicability of the data.
  • Big data platforms need to be able to efficiently store and manage massive amounts of data and support the storage of multi-source heterogeneous data.
  • the data storage of a big data platform can be achieved through Hadoop Distributed File System (HDFS) and the like.
  • HDFS Hadoop Distributed File System
  • Big data platforms can utilize various computing frameworks to perform in-depth data analysis and support complex query and data processing tasks.
  • data processing and computation on a big data platform can be implemented using Apache Spark or Hadoop MapReduce.
  • Apache Spark is a fast and general-purpose big data processing engine that supports batch processing, interactive queries, and stream processing
  • Hadoop MapReduce is a parallel distributed computing model for large-scale datasets.
  • Data Query and Analysis Service The purpose of this service is to extract useful information from massive amounts of data and perform analysis.
  • Common big data query and analysis tools include Apache Spark and Apache HBase.
  • Spark is a fast and general-purpose big data processing engine that supports in-memory computing and provides rich application program interfaces (APIs), which can be used for real-time data processing and complex analysis.
  • APIs application program interfaces
  • HBase is a distributed, column-oriented NoSQL database that is suitable for storing large amounts of structured data and supports real-time query and analysis.
  • big data platforms also need to consider security, scalability, and high performance to ensure stable system operation and reliable data protection.
  • the choice of big data technology stack depends on specific business needs, data scale, real-time requirements, and scalability requirements. Enterprises typically combine and integrate these technologies and tools according to their own circumstances to build solutions suitable for their big data processing needs. With the development of technology, big data systems are becoming increasingly end-to-end, visualized, and intelligent, providing powerful support for enterprise-level applications.
  • Terminal device 102 can be a personal computer device, such as a smartphone, laptop, or desktop computer, or it can be a user-side server; there are no restrictions on this.
  • Server 101 includes various computer devices 1011 that can be servers, such as local servers, cloud servers, or servers in a data center.
  • server 101 can be a database system server, in which case server 101 can provide database-related business services to terminal device 102.
  • different big data technologies are currently used for different data query scenarios. This means that when the latency requirements of multiple data query scenarios need to be met simultaneously, it is necessary to deploy equipment resources according to the needs of multiple big data technologies, resulting in high resource consumption. Furthermore, different big data technologies provide different query engine capabilities based on their own proprietary file formats. For example, the following examples illustrate two different database technology solutions.
  • the first approach Utilizing key-value (K/V) storage and retrieval technology.
  • This scheme uses key-value pairs to store data, with each key corresponding to a unique value. It is well-suited for using the primary key as a query condition. However, when using non-primary keys as query conditions, a full table scan is required, resulting in high query latency.
  • the value of the primary key uniquely identifies each row in a data table; such a column or multiple columns of data are called the primary key of the data table.
  • FIG. 2A is a flowchart illustrating a data query process using key-value (K/V) storage and retrieval technology.
  • data querying is implemented using the combination of the HBase engine and the HFile data format.
  • HBase is a common K/V query engine, while HFile is its proprietary data format. Combining the two enables fast point-to-point or range queries by key.
  • the implementation process of this solution includes the following stages:
  • HBase table creation phase For example, when user data needs to be stored but a corresponding data table does not exist, the HBase engine creates an HBase table. Each HBase table is globally sorted by key, and an HBase table can be horizontally divided into multiple smaller tables, each corresponding to multiple HFile files.
  • S11 Data Loading Phase.
  • the HBase engine in the HBase server needs to store the data as HFile files in the HFile data format to facilitate subsequent data querying.
  • the HBase engine uses the Spark engine to generate HFile files in batches.
  • the Spark engine When a user submits or updates data (as shown in Figure 2A), the Spark engine generates HFile files and sends them to the HBase engine. The HBase engine then sends these files to HDFS for storage. After storage is complete, the corresponding response is returned to the client.
  • the data is sorted by key and divided into multiple blocks (e.g., each block is 64KB in size).
  • the HFile file records the block index, and each block stores data in key/value format; that is, a key must be stored for each value.
  • S12 Data Query Phase.
  • the HBase engine retrieves the data from HDFS based on the user's query request.
  • the mapping between each sub-table and the server storing it is stored through a metadata table.
  • the client if it does not cache the Meta table during the initial query, it needs to request the HBase engine to retrieve and cache the Meta table.
  • the client When retrieving by key, the client first accesses the Meta table to determine the sub-table containing the key, then retrieves the HBase server where the sub-table resides, and subsequently sends a point/range query request to that HBase server to request the required data.
  • the HBase engine on the HBase server reads and parses the HFile file, constructs a B-Tree index structure using the block index, and then locates the block containing the required data based on the B-Tree index structure. Finally, it scans the data in the block and returns the query results, including the key-value pairs in that data block that meet the conditions, to the client.
  • K/V key-value
  • key-based queries i.e., queries using the primary key as the condition
  • this approach is suitable for point-to-point or range query scenarios.
  • multidimensional queries often involve non-key queries (i.e., queries using non-primary keys as conditions).
  • Non-key queries require scanning the entire database tables, resulting in higher latency and failing to meet the latency requirements for multidimensional queries.
  • K/V data structures to store data from relational tables leads to linear expansion of storage space and consumes significant storage resources.
  • the second approach Utilizing online analysis processing (OLAP) technology.
  • OLAP online analysis processing
  • OLAP is a data processing technique used for analyzing and querying large-scale datasets. OLAP analyzes and processes data through multidimensional data models, which consist of multiple dimensions and measures. Suitable data formats for OLAP include Parquet, a columnar storage format.
  • the Spark engine is a multidimensional query engine; combining the Spark engine with Parquet data tables enables efficient OLAP processes, including point-to-point and range queries. See Figure 2B, which illustrates a flowchart of a point-to-point or range query process using the Spark engine and Parquet data format. This process includes the following stages:
  • Parquet tables The creation phase of Parquet data format tables (referred to as Parquet tables).
  • the Spark engine creates Parquet tables, and to accelerate point queries or range queries, the Spark engine partitions/buckets each data table, thus creating partitioned/bucketed Parquet tables.
  • the table file corresponding to a Parquet table is called a Parquet file, and one partition/bucket can correspond to multiple Parquet files.
  • S21 Data Loading Phase.
  • the Spark engine executes the data loading process, writing the data to the corresponding partitions and/or buckets by key (such as frequently queried keys). Within each bucket, the data is sorted by key, ultimately generating the corresponding Parquet file, which is then stored in HDFS.
  • Spark employs a distributed cluster architecture, including Driver and Executor processes, which can be deployed on different device nodes.
  • the data query process requires the cooperation of these components to complete the entire query.
  • the data query process includes the following steps:
  • the Driver process generates the query plan.
  • the Driver process reads the index file from HDFS and uses it to determine the Parquet files that meet the query criteria. Therefore, the final query plan includes a list of the Parquet files to be queried.
  • the process of generating the query plan is a coarse selection of Parquet files that meet the query criteria.
  • the query plan is executed, it is divided into multiple subtasks. Each subtask processes a portion of the Parquet file list, and these subtasks are assigned to multiple Executor processes for execution.
  • Each Executor process executes its own subtask, obtaining data rows that meet the query conditions.
  • an Executor process when an Executor process executes a subtask, it queries the Parquet files assigned to its subtask. Since the Driver process only provides a list of Parquet files and not any index information, the Executor process also reads the footer of each Parquet file in the HDFS list to obtain metadata. Based on the metadata and query conditions, it then retrieves data blocks that meet the query conditions. Afterward, the Executor process reads the data blocks and filters out the data rows that meet the query conditions.
  • the Driver process collects the data rows obtained by the Executor process from executing subtasks and returns them to the client.
  • the Spark engine Since the Spark engine was originally designed to achieve multidimensional data analysis, it provides a large distributed task collaboration framework and a complex query process to meet the computational needs of multidimensional data analysis. However, this results in high latency for key-based point queries/range queries, typically in the order of seconds, which does not meet the latency requirements of point queries or range queries.
  • the first solution cannot recognize the Parquet data format and does not support custom data sources, making it incompatible with multidimensional query requirements.
  • the second solution's data query process is overly cumbersome, resulting in low query efficiency for point or range queries. Therefore, to satisfy both the latency requirements of point or range queries and multidimensional queries, it is necessary to deploy device resources separately according to the needs of various big data technologies. For instance, the database might need to store files in two different data formats simultaneously, which would obviously consume excessive device storage resources, leading to significant resource consumption.
  • this application provides a data query method.
  • this method can determine the information of at least one data block that meets the data query conditions from the data blocks corresponding to the first data table based on the description information of the first data table.
  • This at least one data block is a data block that meets the data query conditions, meaning that at least one data block includes the data required by the first request. Therefore, reading at least one data block can obtain the data requested by the first request.
  • this application allows data tables originally used for data analysis to be used for scenarios such as point queries, range queries, or multidimensional queries. It enables multiple data query scenarios to share a single database technology data table, eliminating the need for separate deployment of storage resources for storing data tables corresponding to different database technologies, as required in related technologies.
  • the description information of the first data table can indicate the data blocks corresponding to the first data table, such as indicating the data included in the data block. Therefore, the data blocks in the first data table that meet the data query conditions can be determined by combining the description information and the data query conditions. This reduces the complexity of the data query process and improves data query performance. For example, compared to the second database technology mentioned above, the embodiments of this application can effectively improve the data query performance of data query services such as point lookup or range lookup based on the data table of data analysis business.
  • FIG. 3 is a schematic diagram of the system architecture of the big data platform provided in this application.
  • This system architecture includes a data query service node 301, a data analysis service node 302, a projection service node 303, and a database 304.
  • Data query service node 301 is used to handle data query services.
  • the data query service retrieves the required data from database 304, where the data is stored.
  • a client can send a data query request to data query service node 301.
  • Data query service node 301 executes the data query process, obtains the corresponding data, and returns a response to the client, which only includes the data retrieved from database 304.
  • data query services can include point-to-point or range-based queries.
  • the data query business node 301 includes a data query engine, which can be called a K/V Server.
  • the K/V Server is a lightweight query engine provided in this application embodiment for processing data query business (such as point query/range query).
  • Data analysis business node 302 is used to handle data analysis tasks.
  • the data analysis task is used to perform data analysis based on the data in database 304. Therefore, when executing a data analysis task, it is actually necessary to first retrieve the corresponding data from database 304, and then perform data analysis based on the retrieved data. The final response returned to the client will contain the results of the data analysis.
  • the data analysis task may include multidimensional data analysis (or OLAP) tasks.
  • Projection service node 303 is used to perform data skipping.
  • Projection service node 303 stores descriptive information for each data table in the database, which indicates the data in the corresponding data blocks of each data table. This descriptive information can also be called index information.
  • Projection service node 303 can preload and store the descriptive information of the data tables through a preloading process, and incrementally update the descriptive information when the data in the data tables changes.
  • data query business node 301 can include two processes: generating a query plan and executing the query plan. When generating the query plan, it can request projection service node 303 to execute the data skipping process.
  • projection service node 303 can determine the data blocks that meet the data query conditions from the corresponding data blocks of the data table to be queried based on its stored descriptive information, and return the information of these data blocks to data query business node 301.
  • Data query business node 301 uses this information to accurately locate and read the data blocks in database 304, and parses the records in these data blocks that meet the data query conditions for the client. This reduces the number of data blocks that the data query business node 301 needs to read and query when executing the query plan, thereby improving data query efficiency.
  • Database 304 is used to store data and metadata.
  • Database 304 serves as a data warehouse for a big data platform, and its data can be stored by one or more storage nodes.
  • Database 304 stores table files corresponding to various data tables; therefore, it can essentially be understood as a file system.
  • database 304 can adopt a distributed storage architecture, meaning it can be a distributed file system.
  • Database 304 may include one or more data tables, which can be used as the foundation for data query and data analysis operations.
  • the data tables in database 304 can be used for point queries, range queries, and multidimensional queries. Therefore, a single set of data in the database can support various different query scenarios.
  • the solution in this application effectively reduces the consumption of data storage resources.
  • This application does not limit the specific type of database 304.
  • the database 304 can store data or metadata under the instruction of the data query business node 301, the data analysis business node 302, or other nodes. It can also transmit data to the data query business node 301 and the data analysis business node 302 under their instruction. For example, when the data query business node 301 finds a data block, it can retrieve the data block from the database 304 based on the data block's address, and the database 304 will then transmit the data block to the data query business node 301.
  • the database 304 can be a computer device or a cluster of computer devices with storage capabilities.
  • the database 304 may include components with storage capabilities within the computer device, such as hard disks or magnetic disks. Any device or apparatus with data storage capabilities is applicable to the embodiments of this application.
  • data tables can be laid out using partitioning and bucketing to store data in database 304.
  • the field used for partitioning is called the partition field.
  • the value of the partition field can be a possible value of an index field in the data table, such as the value of a time field, where a time period can be divided into a partition; or it can be a transformed value of a possible index field in the data table. For example, if an index field in the data table is a user's mobile phone number, then the value of the partition field could be the last digit of the mobile phone number, with each digit corresponding to a partition.
  • a data table corresponds to one or more table files.
  • Each table file can sort data according to a sorting field, and the data blocks within each table file can also be sorted according to the sorting field.
  • Min-Max statistics are recorded for all index fields, and other types of indexes can be created for fields other than the sorting field.
  • the sorting field refers to the index field used to sort the data in the data table. For example, it can be a high-frequency index field, which is the index field with the highest query frequency; a high-frequency index field can also be called a high-frequency query key. Other fields are, for example, second-highest frequency index fields, which are the index fields with the second highest query frequency.
  • Min-Max statistics refer to the minimum and maximum values of an index field.
  • the number of nodes in this system architecture can be one or more.
  • a distributed architecture can be used.
  • multiple data query service nodes 301 can adopt a distributed architecture, allowing client-sent data query requests to be distributed to appropriate data query service nodes 301 based on geographical location, load balancing, and other strategies, thereby improving the response speed of data query requests.
  • multiple projection service nodes 303 can adopt a distributed architecture to implement distributed caching of description information.
  • the multiple projection service nodes 303 can implement distributed caching of description information based on a consistent hashing algorithm, or other algorithms can be used to ensure data consistency among the various projection service nodes 303.
  • the embodiments of this application do not limit the specific form of the data query service node 301, data analysis service node 302, and projection service node 303.
  • These nodes can be hardware devices, such as a single computer device or a cluster of multiple computer devices. Alternatively, these nodes can be virtual machines. Furthermore, these nodes can be a hardware component in a computing device, such as a processor (e.g., a central processing unit (CPU), data processing unit (DPU)), offloading card, accelerator card, etc. Alternatively, these nodes can also be software devices, such as software for managing files, such as query business software, data analysis business software, data skipping software, etc.
  • CPU central processing unit
  • DPU data processing unit
  • offloading card accelerator card
  • these nodes can also be software devices, such as software for managing files, such as query business software, data analysis business software, data skipping software, etc.
  • the data query service node 301, data analysis service node 302, and projection service node 303 can be deployed in different computer devices or in the same computer device; the embodiments of this application do not impose any restrictions on this.
  • the data query service node 301 and the data analysis service node 302 can be deployed in the same computer device.
  • the deployment method of the storage device 200 and the query device 100 is not limited in this embodiment.
  • the storage device 200 and the query device 100 can be deployed in the same computing device; for example, the query device 100 can be a processor (such as a CPU or DPU) in the computing device, and the storage device 200 can be a hard disk in the computing device.
  • the storage device 200 and the query device 100 can also be deployed in different computing devices; for example, the storage device 200 and the query device 100 can be deployed in a storage system, where the query device 100 is a computing node in the storage system (possessing data computing capabilities and undertaking computing tasks in the storage system), and the storage device 200 is a storage node in the storage system (possessing data storage capabilities).
  • the data involved in the data query method is stored in a database.
  • the data in the database can adopt a data format suitable for data analysis, such as Parquet or any other possible data format.
  • a data table needs to be created in the database, and a data loading process needs to be executed.
  • a Parquet table can be created, and a Parquet table file can be generated through the data loading process and stored in the database. Since the table files in the database are in a data format suitable for data analysis, these table files can naturally be applied to data analysis.
  • the data query method of this embodiment provides a low-complexity query process to improve the efficiency and performance of data query based on these table files. This also means that this data query method can achieve the effect of "one set of data" supporting multiple query scenarios, thus eliminating the need to deploy different data formats in related technologies to meet the needs of different query scenarios, which helps to reduce the consumption of storage resources.
  • Figure 4 is a flowchart illustrating the data query method provided in this embodiment. This method can be executed by a second node, such as the data query business node 301 or the data analysis business node 302 shown in Figure 3.
  • a second node such as the data query business node 301 or the data analysis business node 302 shown in Figure 3.
  • Step 401 The second node (K/V server) receives the first request, which is used to request data from the database.
  • the first request includes a query statement.
  • the database includes multiple data tables, which are used for data analysis business.
  • the data analysis business is used to retrieve data from the database and analyze the retrieved data.
  • the first request can be triggered by a user's action on the client.
  • the client can be any client with access to the database; it could be a software client, a small program within a software client, or a webpage in a browser. For example, if a user enters relevant conditions for filtering data while using the client and confirms the request, the client will generate the corresponding query and send the first request to the second node.
  • the first request may be automatically triggered when the user uses the client. For instance, when a user opens a page using the client, and that page needs to display data from the database, the client will send the first request to the second node.
  • the first request may be triggered under other circumstances; this application embodiment does not limit the triggering method of the first request.
  • the types of the first request and the second node can correspond.
  • the first request can be a request corresponding to a data query service
  • the second node can be a data query service node.
  • the first request could be a request for point-to-point or range queries.
  • the first request could also be a request corresponding to a data analysis service
  • the second node could be a data analysis service node.
  • the first request could be a request for OLAP analysis.
  • data query and data analysis services are two different services and can correspond to different service pages.
  • this application embodiment provides an engine for data querying in the second node.
  • This engine can be called a K/V server.
  • the K/V server is a lightweight query engine provided by this application embodiment for processing data query services (such as point queries/range queries). It aims to show that the solution of this application embodiment can enable point queries or range queries to meet the latency requirements of point queries or range queries, just like the K/V technology-based solutions in related technologies.
  • Figure 4 specifically illustrates this as an example.
  • the data query service includes point queries or range queries
  • the types of index fields included in the data query conditions of the query statement included in the first request are usually few, for example, less than or equal to N, where N can be 3 or other values.
  • the database could be a data warehouse for a big data platform.
  • the database includes multiple tables, which are objects used to store data.
  • a table can also be understood as a collection of structured data, a form of structured data organization, typically storing data in rows and columns. Each row represents a record in the table, and each column represents the value of an index field in that record.
  • the first request is used to instruct the retrieval of one or more records from the table based on data query conditions, which typically include the values of one or more index fields in the table.
  • multiple data tables are used for data analysis; in other words, these tables employ data formats suitable for data analysis.
  • the data format of a data table could be Parquet, and the table could also be called a Parquet table or a Parquet table.
  • Parquet is an efficient columnar storage file format, commonly used in big data processing and analysis scenarios, such as the solution in related technology 2 above.
  • the combination of the Spark engine and Parquet tables enables efficient OLAP analysis. Compared to data tables based on key-value storage technology, Parquet tables occupy less storage space.
  • a drawback of Parquet tables is that their performance for point-to-point or range queries cannot meet the latency requirements of these queries.
  • the solution in this embodiment improves the query efficiency of data query operations such as point-to-point or range queries by using Parquet tables, thereby increasing the probability of meeting the latency requirements of point-to-point or range queries. Specific details will be provided later.
  • the solution in this application reuses data originally used for data analysis to improve data query efficiency.
  • each piece of data stored in the database can be used for both data analysis and data query (such as point query or range query), achieving the effect that one piece of data supports multiple query scenarios, or that multiple query scenarios can use the same big data technology data table, thereby effectively saving the resources required for data storage.
  • the first request may describe the query statement in JavaScript object notation (JSON) format.
  • JSON JavaScript object notation
  • other formats may be used, and this application embodiment does not limit this.
  • Step 402 The second node (K/V server) determines the information of at least one data block based on the description information of the first data table.
  • the at least one data block is a data block in the data block corresponding to the first data table that meets the data query conditions.
  • the first data table is determined based on the first request, and the description information is used to indicate the data block corresponding to the first data table.
  • the first request includes a query statement.
  • the query statement By parsing the query statement, the data table to be queried and the data query conditions can be determined.
  • the first data table is the data table determined based on the query statement.
  • the query statement in the first request can be parsed to determine the first data table to be queried.
  • the query statement includes at least one query command and clauses.
  • the query command also known as a query function, is used to specify which data to return and how to filter, sort, and combine the data. For example, if the data query conditions specify one or more data tables, then those one or more data tables are the first data table. Alternatively, if the data query conditions do not specify a data table, then all data tables in the database are the first data table.
  • Description information is used to indicate the data in the data blocks corresponding to each data table in the database. Based on this description information, the data blocks in the data blocks corresponding to the first data table that meet the data query conditions can be determined, thus reducing the number of data blocks that need to be read. Description information can also be called index information or an index.
  • one data table corresponds to one set of description information, which indicates the data in the data blocks corresponding to that data table.
  • the data of a data table is stored in one or more table files, and one table file can include one or more data blocks.
  • a data block can include the values of one or more index fields, and the values of the index fields are essentially the data in the data table.
  • the data blocks corresponding to a data table are the data blocks included in the table file corresponding to that data table.
  • the description information is used to indicate the range of values for the data in a data block. It can also be described as indicating the range of values for the index fields included in the data block. This range of values can help filter data blocks in which the data to be queried is not located, thereby reducing the number of data blocks that need to be read.
  • Step 403 The second node (K/V server) reads at least one data block from the database based on information from at least one data block.
  • At least one data block includes the data requested in the first request.
  • the information for each data block may include one or more of the following: the identifier of the table file to which the data block belongs, the address offset of the data block within the file, or the length of the data block.
  • the identifier of the table file can be the name or ID of the table file, etc.
  • the database can be implemented using HDFS.
  • HDFS Based on information from at least one data block, at least one data block can be read from HDFS.
  • the second node can call HDFS's seek and read interfaces.
  • the seek interface is called with the offset as a parameter
  • the read interface is called with the length as a parameter.
  • HDFS will send the data block specified by the offset and length to the second node.
  • HDFS sends the data block as a binary stream, and the second node can combine the metadata of the data block to construct a data block from the binary stream.
  • Spark first needs to generate a query plan in the Driver process.
  • the Driver process reads the index file from HDFS and determines the Parquet files that meet the query conditions based on this index file. Therefore, the final generated query plan only includes a list of Parquet files to be queried; the process of generating the query plan is essentially a coarse selection of Parquet files that meet the query conditions.
  • the query plan is executed, it is divided into multiple subtasks. Each subtask processes a portion of the Parquet files in the list, and these subtasks are assigned to multiple Executor processes for execution. Each Executor process executes its own subtask, obtaining data rows that meet the query conditions.
  • an Executor process When an Executor process executes a subtask, it queries based on the Parquet files assigned to its subtask. Since the Driver process only provides a list of Parquet files and not any index information, the Executor process also reads the footer of each Parquet file in the list from HDFS to obtain metadata. Then, based on the metadata and query conditions, it retrieves the data blocks that meet the query conditions. This is equivalent to needing to interact with HDFS twice, and the filtering process also needs to be executed twice. This is due to Spark's general distributed architecture, which results in extremely low query efficiency and high latency when used for point queries or range queries.
  • the solution in this application preloads description information to pre-store index information scattered in the table files.
  • This description information can indicate the data blocks of each table file, so that the data blocks that meet the data query conditions can be directly located based on the description information, without the need for multiple interactions and data filtering processes as in the related technologies mentioned above. It can be seen that the query process of the solution in this application is less complex, thus the query efficiency is higher and the data query latency is lower.
  • Step 404 The second node (K/V server) filters at least one data block for records that meet the data query conditions and sends the records that meet the data query conditions to the client.
  • records that meet the data query conditions can be filtered out, with each record representing a row in the first data table. Then, the data from the records that meet the query conditions is sent to the client.
  • the data returned to the client may include all the data from each record, or it may include data from some indexed fields of each record.
  • At least one data block that meets the data query conditions can be determined from the data blocks corresponding to the first data table based on the description information of the first data table.
  • This at least one data block is a data block that meets the data query conditions; that is, at least one data block includes the data required by the first request. Therefore, reading at least one data block can obtain the data requested by the first request.
  • this embodiment allows data tables originally used for data analysis to be used for scenarios such as point queries, range queries, or multidimensional queries. This allows multiple data query scenarios to share data tables using the same database technology, eliminating the need for different database technologies and thus avoiding the need to deploy resources separately for multiple database technologies. Instead, resources can be deployed for a shared database technology, reducing resource overhead.
  • the description information of the first data table can indicate the data blocks corresponding to the first data table, such as the data included in the data block. Therefore, the second node can combine this description information and the data query conditions to determine the data blocks in the first data table that meet the data query conditions. This reduces the complexity of the data query process and improves data query performance.
  • Figure 5 is a schematic diagram of another data query method provided in this application embodiment.
  • the difference from the process shown in Figure 4 is that after receiving the first request, the second node needs to parse the query statement in the first request to extract the first data table to be queried, the data query conditions, and the index fields to be queried.
  • the first data table to be queried refers to the range of data to be queried; when the first data table is specified in the query statement, only that first data table will be queried.
  • the data query conditions also known as data filtering conditions, are used to filter data rows in the first data table that meet specific conditions.
  • the index fields to be queried indicate the index fields corresponding to the data to be returned to the client.
  • This data query method includes the following steps:
  • Step 501 The second node (K/V server) receives the first request, which is used to request data from the database.
  • the first request includes a query statement.
  • Step 502 The second node (K/V server) parses the query statement in the first request to determine the first data table and data query conditions.
  • the query statement includes one or more of the following:
  • the FROM clause specifies the identifier of the data table to be queried. This identifier can be the table's ID or name.
  • the first data table is the data table specified by the FROM clause.
  • the data table specified by the FROM clause can include one or more.
  • the number of first data tables is also multiple, or in other words, the first data tables include multiple data tables. In the case of multiple first data tables, each of them can be queried using subsequent methods. Therefore, the following introduction mainly uses a single first data table as an example, and will not be elaborated further. For example, if the database includes a data table named "employees", then a SELECT command with the clause "FROM employees" is used to specify that data should be queried from the employees table.
  • the SELECT command is used to select data to be retrieved from the database.
  • the employees table includes the following: id, name, position and salary, each of which is an index field.
  • a SELECT command would be "SELECT name, salary”. This command means that the data in the name and salary columns will be retrieved. In other words, the data returned to the client will include the data in the name and salary columns.
  • query statements can also include other commands without restriction.
  • the first request includes the following query statement:
  • This command retrieves the name and salary columns for employees whose position is Manager from the employees table.
  • the second node After receiving the first request, the second node will parse the query statement included in the first request to obtain the first data table to be queried, the data query conditions, and the index fields to be queried.
  • Step 503 The second node (K/V server) determines the information of at least one data block based on the description information of the first data table.
  • the at least one data block is a data block in the data block corresponding to the first data table that meets the data query conditions.
  • the description information is used to indicate the data block corresponding to the first data table.
  • Step 504 The second node (K/V server) reads at least one data block from the database based on information from at least one data block.
  • Step 505 The second node (K/V server) filters at least one data block for records that meet the data query conditions and sends the records that meet the data query conditions to the client.
  • parsing the query statement in the first request can also reveal the index field to be queried. Then, the data corresponding to the index field to be queried from the records that meet the data query conditions can be returned to the client.
  • this application embodiment provides a first node that provides a dedicated Data Skipping service.
  • the first node is, for example, the projection service node shown in Figure 3.
  • the first node needs to store the description information of multiple data tables included in the database.
  • Figure 6 is a flowchart illustrating the process of the first node obtaining description information.
  • the first node can obtain description information in the following two ways, including but not limited to:
  • Method 1 The first node preloads the description information of the data table from HDFS.
  • the first node can execute step 611, which reads the table file from HDFS, and then execute step 612, which determines and stores the description information of multiple data tables based on the table file corresponding to each data table. For example, upon initial use, the first node can obtain the description information from the table file in HDFS. Alternatively, the first node can periodically read the table file from HDFS to obtain the description information.
  • the first node can preload the descriptive information scattered in the multidimensional analysis file (i.e., the table file) into itself, so that when the first node provides the Data Skipping service to the second node, it can use this descriptive information to improve the data query efficiency.
  • the multidimensional analysis file i.e., the table file
  • the first node can update the storage description information based on the updates to HDFS.
  • the big data platform also includes a third node, which can be used to maintain HDFS.
  • the third node can be a data analysis business node (such as a Spark engine node) as shown in Figure 3.
  • the third node can update the data in HDFS based on operations on HDFS.
  • the third node can also notify the first node to update the description information.
  • the third node can send a third request to the first node.
  • the first node receives the third request, which instructs the third node to update the second data table.
  • the first node executes step 622, which reads the table file of the second data table from HDFS according to the third request, and executes step 623, which determines and stores the description information of the second data table based on the table file of the second data table.
  • the software development kit (SDK) deployed on the first node can be called from the third node.
  • the third node updates HDFS it can call the SDK to send a third request to the first node.
  • the third request could include the operation performed by the third node on HDFS and the object of that operation, such as inserting, deleting, or modifying data. This allows the first node to determine the second data table based on the operation and the object, and then update the description information of the second data table accordingly.
  • the first node determines the description information of the second data table. This could be done by reading the complete table file of the second data table and redetermining its description. Alternatively, the first node could read only a portion of the data, a portion of the table file, or a portion of the data blocks, and update the description information of the second data table based on the read content, thus achieving incremental updates to the description information of the second data table. Incremental updates reduce the amount of data transmission, lower the load on the first node, and allow more resources to be used for data skipping, which helps improve data query efficiency.
  • the Data Skipping process can be performed based on this information.
  • the time consumed by the Data Skipping process varies for different tables.
  • the second node requests the first node to perform Data Skipping, there is interaction overhead between the first and second nodes. If this interaction overhead accounts for a large portion of the network round-trip time (RTT), while the Data Skipping process only occupies a small portion of the RTT, then setting up the first node alone no longer offers any advantage.
  • RTT network round-trip time
  • this application provides a scheme for adaptively selecting local or remote Data Skipping based on the size of the table.
  • Local Data Skipping refers to the second node performing the Data Skipping
  • remote Data Skipping refers to the second node requesting the first node to perform the Data Skipping process.
  • the size of the table primarily refers to the amount of data in the table. When the amount of data in a table exceeds a first threshold, the table is considered a large table; when the amount of data in a table is less than or equal to the first threshold, the table is considered a small table.
  • the value of the first threshold can be set according to actual conditions, such as an empirical value or a value determined experimentally.
  • the small table is referred to as a first type of data table
  • the large table is referred to as a second type of data table.
  • Figure 7 is another flowchart illustrating the data query method provided in the embodiments of this application.
  • Step 701 The client sends a first request to the second node (K/V server).
  • the second node (K/V server) receives the first request, which is used to request data from the database.
  • the first request includes a query statement.
  • Step 702 The second node (K/V server) parses the query statement in the first request to determine the first data table and data query conditions.
  • Step 703 The second node (K/V server) determines whether the description information of the first data table has been stored.
  • Step 704 If the judgment result of step 703 is yes, that is, if the second node (K/V server) has stored the description information of the first data table, the second node (K/V server) determines the information of at least one data block based on the description information of the first data table.
  • at least one data block is a data block in the data block corresponding to the first data table that meets the data query conditions.
  • the second node When the first data table is a type 1 data table, the second node stores the description information of the first data table. The second node then executes a local Data Skipping process, which determines at least one data block that meets the data query conditions based on the stored description information.
  • Type 1 data tables are the small tables mentioned above. The Data Skipping process for this type of data table takes less time. If the first node is requested to execute the Data Skipping process, the interaction overhead between the second node and the first node accounts for a large portion of the RTT (Round-Trip Time).
  • the second node executes the Data Skipping process itself, because the first data table is small, the resource consumption is relatively small, and the interaction overhead with the first node can be saved, reducing the duration of the Data Skipping process for the first data table and thus improving data query efficiency.
  • the second node can send a request to the first node for the description information of the first data table.
  • the second node can read the table file corresponding to the first data table from the database and then obtain the description information of the first data table based on the read table file.
  • Step 705 If the judgment result of step 703 is negative, that is, if the second node (K/V server) does not store the description information of the first data table, the second node (K/V server) sends a second request to the first node (projection service node).
  • the second request includes the identifier of the first data table and the data query conditions.
  • the second node does not store the description information of the first data table in the following situations:
  • Scenario 1 If the first data table is a second type of data table, then the second node will not store the description information of this type of data table.
  • the first type of data table is the large table mentioned above.
  • the data skipping process for this type of data table takes a long time and will consume a lot of device resources of the second node. Therefore, in order to avoid affecting the execution of other functions in the second node, the data skipping process can be executed by requesting a dedicated first node.
  • Scenario 2 The second node queries the first data table for the first time.
  • the second node queries the first data table for the first time it does not perform a type check on the first data table beforehand, therefore the second node does not store the descriptive information of the first data table.
  • Step 706 The first node (projection service node) determines the information of at least one data block based on the description information of the stored first data table.
  • Step 707 The first node (projection service node) sends a first response to the second node (K/V server).
  • the second node receives the first response from the first node.
  • the first response includes information about at least one data block.
  • the second node when the second node does not store the description information of the first data table, the second node will execute a remote data skipping process, that is, send a second request to the first node to request the first node to execute data skipping.
  • the first node will determine at least one data block that meets the data query conditions based on the description information of the first data table it stores.
  • Step 708 The second node (K/V server) reads at least one data block from the database based on information from at least one data block.
  • Step 709 The second node (K/V server) filters records in at least one data block that meet the data query conditions.
  • Step 710 The second node (K/V server) sends the records that meet the data query conditions to the client.
  • Figure 8 shows a flowchart illustrating the process of the second node storing the descriptive information of the first data table.
  • Step 801 The first node (projection service node) sends information for the first duration to the second node (K/V server).
  • the second node (K/V server) receives the information for the first duration from the first node (projection service node).
  • the first duration is the time consumed by the first node to determine at least one data block, that is, the time consumed by the first node to execute the Data Skipping process.
  • the first duration information could be included in the first response.
  • the first node could also send the first duration information to the second node before or after sending the first response to the second node.
  • Step 801 is an optional step, and therefore is shown as a dashed line in Figure 8.
  • Step 802 The second node (K/V server) determines that the type of the first data table is the first type.
  • the second node can determine whether the type of the first data table is type one based on the amount of data corresponding to the first data table. If the amount of data corresponding to the first data table is less than or equal to a first threshold, the type of the first data table is type one; otherwise, if the amount of data corresponding to the first data table is greater than the first threshold, the type of the first data table is type two.
  • the second node can also determine whether the type of the first data table is the first type based on the information of the first duration. If the first duration is less than or equal to the second threshold, the type of the first data table is determined to be the first type; otherwise, if the first duration is greater than the second threshold, the type of the first data table is determined to be the second type.
  • the second threshold can be a fixed value, such as a fixed value determined empirically or experimentally. Alternatively, the second threshold can be determined based on the first round-trip time (RTT), for example, the second threshold can be the product of the first RTT and a first ratio. The value of the first ratio can be configured according to requirements.
  • the first RTT is the duration between the sending time of the second request and the receiving time of the first response.
  • the sending time can be the start time of the second request, the end time of the second request, or the midpoint between the two
  • the receiving time can be the start time of the first response, the end time of the first response, or the midpoint between the two, etc.
  • the first RTT is the RTT when the second node requests the first node to perform Data Skipping.
  • Step 803 The second node (K/V server) sends a request to the first node (projection service node) to obtain the description information of the first data table.
  • Step 804 The first node (projection service node) sends the description information of the first data table to the second node (K/V server).
  • Step 805 The second node (K/V server) stores description information.
  • the Data Skipping process performed on the second or first node, which is the process of determining information about at least one data block based on the description information of the first data table.
  • the Data Skipping process includes two steps: partition (and/or bucketing) pruning and data pruning.
  • Partition (and/or bucketing) pruning selects results from the cached file list that match the partition (and/or bucketing) conditions included in the tree query conditions.
  • Data pruning, based on the partition (and/or bucketing) pruning results checks each table file to see if it meets the data query conditions; if it does, the table file is retained; otherwise, it is excluded.
  • Spark generates a query plan in the Driver process.
  • This query plan provides a list of table files that meet the data query conditions, and then distributes it to the distributed Executor processes.
  • the distributed Executor processes scan the list of files provided by the query plan. Since the Executor processes do not include descriptive information, they also need to read information such as the page footer of the table files from the database before determining the data blocks that meet the conditions. This means that the descriptive information needs to be loaded twice, and the Data Skipping process also needs to be executed twice. This is determined by Spark's general distributed architecture, resulting in extremely low query efficiency and high latency when used for point queries or range queries.
  • the description information of the first data table may include one or more of the following:
  • Metadata also known as key metadata, indicates the set or list of indexed fields included in the first data table. This metadata also indicates the first indexed field in the first data table.
  • the first data table comprises multiple partitions, which are determined based on the value of the first indexed field.
  • the first indexed field may include a partition field and/or a bucketing field.
  • a data table (such as the first data table) includes one or more partitions.
  • the partitions of a data table can be determined by the value of one of the index fields in the data table, or its transformed value. This index field is used to divide the data table into partitions; therefore, it can be called the partition field.
  • Different partitions of the data table correspond to different values or ranges of values in the partition field. For example, if the partition field is time, each partition of the data table can include data for a continuous time period.
  • a partition can be understood as a storage area; different partitions of a data table can correspond to different storage areas, and one partition can correspond to one storage area.
  • a partition can be understood as a sub-table of a data table.
  • the partitioning field can be the same or different for different data tables.
  • the partitioning field can be any index field in the data table, or a transformation of any index field.
  • the partitioning field can be a specific field in a given data table, or it can be a high-frequency index field (the index field with the highest query frequency).
  • a data table can then be horizontally partitioned into different partitions (or storage locations) based on the value of the high-frequency index field (or its transformed value, such as a hash value).
  • the high-frequency index field can remain unchanged after being determined.
  • the high-frequency index field can be determined empirically from the table's index fields, and once it is chosen as the partitioning field, the partitioning field of the data table will not change.
  • the high-frequency index field of a data table can also change, and the partitioning of the data table can change as the high-frequency index field changes.
  • a partition of a data table can include one or more buckets. Buckets are determined by the value of one of the index fields in the data table. This index field is used to divide the data table into buckets, and is therefore also called the bucket field. Different buckets correspond to different value ranges of the bucket field. For example, if the bucket field is time, the data table is split into different buckets according to time. Each bucket includes data for a continuous time period, and each bucket can include data for a continuous sub-time period. In some examples, buckets can also be understood as storage areas; different buckets within a partition can correspond to different sub-storage areas, with one bucket corresponding to one sub-storage area.
  • the bucketing field can be any index field in the data table.
  • the partitioning field can be a specific field in the data table, or it can be a high-frequency index field, a second-highest frequency index field, or other fields in the data table.
  • the second-highest frequency index field refers to the index field with the second highest query frequency; it can also be called the second-highest query key. Therefore, based on the value of the high-frequency or second-highest frequency index field (or its transformed value, such as a hash value), the data in the data table is horizontally split into different storage locations. If used in conjunction with partitioning, the bucketing field is used to split the data within each partition.
  • the first indication information can also be called the partition file list or partition file set.
  • the first indication information is used to indicate the table files corresponding to the multiple partitions.
  • the first data table includes multiple partitions, and the table file corresponding to the first partition includes the data corresponding to the first partition.
  • the first partition can be any of the multiple partitions.
  • One of the table files corresponding to the multiple partitions includes one or more data blocks, and one of the data blocks includes the values of one or more index fields.
  • the first indication information can indicate the table file corresponding to each partition in the multiple partitions of the first data table.
  • the second indication information can also be called the min-max statistics of the table file or the min-max statistics of the data block.
  • the second indication information is used to indicate the value range of the index fields of the data blocks included in one of the table files corresponding to multiple partitions.
  • the value range can be characterized by min-max statistics.
  • the min-max statistics of a data block indicate the minimum and maximum values in the data block. Since the data in the data block is sorted according to the values of the sort field, the values of the index fields in the data block are all within the range of the minimum and maximum values.
  • a data block may include the values of one or more index fields. When a data block includes the values of multiple index fields, the min-max statistics of the data block may also include the min-max statistics of multiple index fields.
  • the sort field refers to the index field used to sort the data in the data table, such as the high-frequency query key, the second-highest frequency index field, etc.
  • the table file corresponding to the first data table can be a Parquet file.
  • a Parquet file contains one or more data blocks (each data block is 128MB or larger), and each data block stores multiple data rows column-wise.
  • the Parquet file also records the min-max statistics for each field in each data block.
  • a Bloom filter can be used to help filter out table files or data blocks that do not contain the data to be queried.
  • a Bloom filter can refer to a Bloom filter on one or more indexed fields of the first data table.
  • Bitmap can also be used to help determine data blocks.
  • the description information may also include information in any other possible indexing methods, which is not limited in this embodiment of the application.
  • the data query criteria include the value of at least one indexed field.
  • the process for determining at least one data block that meets the data query criteria is described below. Please refer to Figure 9, which is a flowchart illustrating the process of determining information for at least one data block. This process can be executed by either the first node or the second node.
  • Step 901 Determine whether at least one index field includes the first index field based on the metadata.
  • the first index field may include a partition field and/or a bucket field.
  • Figure 9 illustrates this using the partition field as the first index field as an example.
  • Step 902 If the determination result of step 901 is yes, that is, it is determined that at least one index field includes the first index field, then determine the second partition from the multiple partitions of the first data table.
  • the second partition is the partition where the value of the first index field in at least one index field is located.
  • the partition containing the first time period can be determined from the multiple partitions of the first data table.
  • Step 903 Determine the table file corresponding to the second partition based on the first instruction information.
  • the first indication information also known as the partition file list, indicates the table files corresponding to each partition of the first data table. After determining the second partition, the table files corresponding to the second partition can be determined based on the first indication information, thus narrowing down the scope of the data query.
  • Step 904 Determine information for at least one data block based on the second instruction information.
  • the at least one data block includes values for one or more index fields from at least one index field.
  • the second instruction information can indicate the value range of the data blocks included in the table file corresponding to the first data table. Then, based on the value range of the data blocks included in the table file corresponding to the second partition in the second instruction information, and the value of at least one index field included in the data query conditions, at least one data block that satisfies the data query conditions can be determined, and information about at least one data block can be obtained.
  • the value range is min-max statistics. For each data block to be queried, the data block can be filtered based on whether the value of each index field in at least one index field is between the min-max values. If it is not between the min-max values, the data block may include the data to be queried.
  • Figure 10 is a schematic diagram of a process for determining a data block provided in an embodiment of this application.
  • the min-max information of index field c1 is [‘a’,‘e’] and [‘x’,‘y’] respectively.
  • file1 contains two data blocks, namely data block 1 and data block 2.
  • the min-max information of index field c1 in the two data blocks is [‘a’,‘b’] and [‘c’,‘e’] respectively.
  • the data pruning result is file1 ('d'>'a' and 'd' ⁇ 'e'), and file2 is excluded (because 'x'>'d' causes the query condition to not be met).
  • the Executor process reads the footer of file1 from HDFS. Based on the index information saved in the footer and combined with the query conditions, it is determined that only data block 2 meets the conditions. Finally, the Driver process reads data block 2.
  • the K/V server can directly determine that data block 2 meets the data query conditions, so data block 2 can be read directly.
  • Step 905 If the judgment result of step 901 is negative, that is, it is determined that at least one index field does not include the first index field, then the table file corresponding to the first data table is determined according to the first indication information. In other words, when at least one index field does not include the partition field or bucket field, partitions cannot be filtered, and the data blocks included in the table files of all partitions of the first data table will be queried.
  • Step 906 Determine information for at least one data block based on the second instruction information.
  • the at least one data block includes values for one or more index fields from at least one index field.
  • step 904 based on the value range of the data blocks included in the table files of all partitions of the first data table, and the value of at least one index field included in the data query conditions, at least one data block that satisfies the data query conditions can be determined, and information about at least one data block can be obtained.
  • this application embodiment provides a lightweight query engine K/V server.
  • This K/V server receives point/range query requests from clients, parses them to the query table and query conditions, and then performs a data skipping process on the table files of the data table based on multidimensional data analysis. This process identifies data blocks that meet the data query conditions, reducing the number of data blocks that need to be read. Finally, the data blocks are read to obtain the query results.
  • the device hosting the query engine faces a process memory limit issue.
  • This application embodiment sets up a dedicated data skipping service node, i.e., the first node, to alleviate the device memory limit problem.
  • a scheme of adaptively selecting local or remote data skipping for tables of different sizes is adopted, balancing interaction overhead and data skipping resource consumption.
  • this data query method can perform efficient point/range queries based on data analysis business (such as OLAP analysis).
  • This method implements a single data table format that supports multiple data query scenarios (such as point queries, range queries, or data analysis business). Only one set of data is needed.
  • this method can save nearly half of the storage resources and reduce data management costs.
  • the data query method in this application embodiment has lower query process complexity and finer-grained results when performing data skipping (files in related technologies, data blocks in this application embodiment).
  • the query process takes less time and is more efficient, for example, performing better query performance for point or range queries, approaching the query performance of key-value (K/V) query schemes in related technologies.
  • K/V key-value
  • this application embodiment also provides a data query device, which is used to execute the data query method in the above method embodiments.
  • the data query device 110 includes a receiving module 1101, a determining module 1102, and a data reading module 1103. Specifically, in the data query device 110, the modules are connected through a communication path.
  • the receiving module 1101 is used to receive a first request, which requests to retrieve data from a database.
  • the first request includes a query statement.
  • the database includes multiple data tables used for data analysis operations, which retrieve and analyze data from the database.
  • the determining module 1102 is used to determine information about at least one data block based on the description information of the first data table.
  • the at least one data block is a data block in the data blocks corresponding to the first data table that meets the data query conditions.
  • the first data table and the data query conditions are determined based on the first request.
  • the description information indicates the data blocks corresponding to the first data table.
  • the data reading module 1103 is used to read at least one data block from the database based on the information of the at least one data block.
  • the determining module 1102 is specifically used to determine information of at least one data block based on the stored description information; or, in the absence of stored description information, to send a second request to the first node and receive a first response from the first node, wherein the second request includes the identifier of the first data table and data query conditions, and the first response includes information of at least one data block.
  • the information of at least one data block includes one or more of the following: the identifier of the table file to which each data block belongs; the address offset of each data block in the table file to which it belongs; or, the length of each data block.
  • the determining module 1102 is further configured to obtain and store description information from the first node when the type of the first data table is a first type; wherein the amount of data corresponding to the data table of the first type is less than or equal to a first threshold.
  • the receiving module 1101 is further configured to receive information about a first duration from the first node, the first duration being the duration consumed by the first node in determining at least one data block.
  • the determining module 1102 is further configured to determine that the type of the first data table is a first type if the first duration is less than or equal to a second threshold.
  • the second threshold is determined based on the first round-trip time, which is the duration between the sending time of the second request and the receiving time of the first response.
  • the description information includes one or more of the following: metadata, which indicates a first index field included in the first data table, wherein the first data table includes multiple partitions, the multiple partitions being determined based on the value of the first index field; first indication information, which indicates the table files corresponding to the multiple partitions respectively, wherein the table file corresponding to the first partition among the multiple partitions includes the data corresponding to the first partition, and one of the table files corresponding to the multiple partitions includes one or more data blocks, one of the data blocks including the value of one or more index fields; and second indication information, which indicates the value range of the index field of the data block included in one of the table files corresponding to the multiple partitions.
  • the data query conditions include the value of at least one index field.
  • the determining module 1102 is specifically configured to: determine, based on metadata, that at least one index field includes a first index field; determine a second partition from multiple partitions, the second partition being the partition where the value of the first index field in at least one index field is located; determine the table file corresponding to the second partition based on first indication information; and determine information about at least one data block based on second indication information, the at least one data block including the value of one or more index fields in at least one index field.
  • the data query conditions include the value of at least one index field.
  • the determining module 1102 is specifically configured to: determine, based on metadata, that at least one index field does not include the first index field; determine, based on first indication information, the table file corresponding to the first data table; and determine, based on second indication information, information about at least one data block, wherein the at least one data block includes the value of one or more of the at least one index field.
  • the determining module 1102 is further configured to determine data rows in the at least one data block that meet the data query conditions.
  • the data query device 120 includes a receiving module 1201, a determining module 1202, and a sending module 1203. Specifically, in the data query device 120, the modules are connected through a communication path.
  • the receiving module 1201 is configured to receive a second request from a second node, the second request including an identifier of a first data table and data query conditions; wherein, the first data table is included in a database, the database includes multiple data tables, the multiple data tables are used for data analysis operations, and the data analysis operations are used to retrieve data from the database and analyze the retrieved data.
  • the determining module 1202 is configured to determine, based on the stored description information of the first data table, at least one data block in the data blocks corresponding to the first data table that meets the data query conditions, the description information being used to indicate the data block corresponding to the first data table.
  • the sending module 1203 is configured to send a first response to the second node, the first response including first indication information, the first indication information being used to indicate at least one data block.
  • the data query device 120 further includes a description information loading module 1204.
  • the description information loading module 1204 is used to: in response to a pre-loading instruction, read table files corresponding to multiple data tables in the database, and determine description information for the multiple data tables based on the table files; or, in response to a third request from a third node, read a table file corresponding to a second data table, and determine description information for the second data table based on the table file, wherein the third request is used to instruct the third node to update the second data table, and the third node is used to maintain the database.
  • the information of at least one data block includes one or more of the following: the identifier of the table file to which each data block belongs; the address offset of each data block in the table file to which it belongs; or, the length of each data block.
  • the sending module 1203 is also used to send information about a first duration to the second node, the first duration being the duration consumed by the first node in determining at least one data block.
  • the descriptive information includes one or more of the following: Metadata, used to indicate a first index field included in the first data table, wherein the first data table includes multiple partitions, the multiple partitions being determined based on the value of the first index field; First indication information, used to indicate the table files corresponding to the multiple partitions respectively, wherein the table file corresponding to the first partition among the multiple partitions includes the data corresponding to the first partition, and one of the table files corresponding to the multiple partitions includes one or more data blocks, one of the data blocks including the value of one or more index fields; Second indication information, used to indicate the value range of the index field of the data block included in one of the table files corresponding to the multiple partitions.
  • Metadata used to indicate a first index field included in the first data table, wherein the first data table includes multiple partitions, the multiple partitions being determined based on the value of the first index field
  • First indication information used to indicate the table files corresponding to the multiple partitions respectively, wherein the table file corresponding to the first partition among the multiple partitions includes the
  • the data query conditions include the value of at least one index field.
  • the determining module 1202 is specifically configured to: determine, based on metadata, that at least one index field includes a first index field; determine, from multiple partitions, a second partition that is the partition where the value of the first index field in the at least one index field is located; determine, based on first indication information, the table file corresponding to the second partition; and finally, based on second indication information, determine information about at least one data block, where the at least one data block includes the value of one or more index fields in the at least one index field.
  • the data query conditions include the value of at least one index field.
  • the determining module 1202 is specifically configured to: determine, based on metadata, that at least one index field does not include the first index field; determine, based on first indication information, the table file corresponding to the first data table; and finally, based on second indication information, determine information about at least one data block, wherein the at least one data block includes the value of one or more of the at least one index field.
  • the module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible.
  • the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical units, or be integrated into a single module.
  • the integrated modules can be implemented in hardware or as software functional modules. In the illustrations of Figures 11 and 12, the units represented by dashed lines are optional functional units.
  • the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, mobile phone, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.
  • the aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
  • the data query device 130 provided in this application may have the structure shown in FIG13, including a processor 1301 and a memory 1302 connected to the processor 1301.
  • the processor 1301 and the memory 1302 can be interconnected via a bus.
  • the processor 1301 can serve as the main processor of the data query device 130, i.e., the control core of the data query device 130.
  • the bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • the bus may be divided into an address bus, a data bus, a control bus, etc.
  • the specific connection medium between the processor 1301 and the memory 1302 is not limited in the embodiments of this application.
  • Memory 1302 may include the aforementioned main memory.
  • Memory 1302 may be volatile memory, such as random access memory; it may also be non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1302 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.
  • ROM read-only memory
  • HDD hard disk drive
  • SSD solid-state drive
  • Memory 1302 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.
  • Memory 1302 may be a combination of the aforementioned memories. Memory 1302 may be used to store software programs and modules.
  • the processor 1301 may include one or more processors.
  • the processor 1301 executes various functional applications and data processing of the data query device 130 by running software programs and modules stored in the memory 1302, such as the data query method provided in the embodiments of this application.
  • the structures illustrated in the embodiments of this application do not constitute a specific limitation on the data query device.
  • the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements.
  • the illustrated components may be implemented in hardware, software, or a combination of software and hardware.
  • the method steps in the embodiments of this application can be implemented in hardware or by a processor executing a computer program or instructions.
  • the computer program or instructions can constitute a computer program product.
  • Embodiments of this application also provide a computer program product comprising computer-executable instructions.
  • the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments.
  • Computer-executable instructions can be stored in a computer-readable storage medium.
  • This application also provides a computer-readable storage medium storing executable instructions.
  • the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.
  • the computer-readable storage medium may be random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, portable hard disk, CD-ROM, or any other form of computer-readable storage medium known in the art.
  • RAM random access memory
  • ROM read-only memory
  • PROM programmable read-only memory
  • EPROM erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • register hard disk, portable hard disk, CD-ROM, or any other form of computer-readable storage medium known in the art.
  • Computer-executable instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
  • the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.
  • the computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media.
  • the available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive.
  • This application also provides a computer program product, including instructions that, when run on a computer, cause the computer to execute the method performed by the first node or the second node.
  • This application provides a chip system including a processor and an interface.
  • the processor is used to call and execute instructions from the interface, causing the chip system to implement the methods described in the above embodiments.
  • This chip system may be composed of chips or may include chips and other discrete devices.
  • This application also provides a chip, including a processor, for supporting the communication device in implementing the functions involved in the first node or the second node in the above method embodiments.
  • the chip is connected to a memory or the chip includes a memory for storing computer programs or instructions and data necessary for the communication device.
  • the disclosed systems, apparatuses, and methods can be implemented in other ways.
  • the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods.
  • multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
  • the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
  • the units described as separate components may or may not be physically separate.
  • the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
  • the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
  • the aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), RAM, magnetic disks, or optical disks.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种数据查询方法与设备，用于实现一种数据表支持多种数据查询场景的需求，提升数据库的数据查询性能。例如，该数据查询方法包括：接收第一请求，第一请求用于请求获取数据库中的数据，第一请求包括数据查询条件；其中，数据库包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。根据第一数据表的描述信息确定至少一个数据块的信息，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块，第一数据表是根据第一请求确定的，描述信息用于指示第一数据表对应的数据块。根据至少一个数据块的信息，从数据库中读取至少一个数据块。

Description

一种数据查询方法与设备
相关申请的交叉引用
本申请要求在2024年06月27日提交中华人民共和国国家知识产权局、申请号为202410855954.4、申请名称为“一种数据查询方法与设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及大数据技术领域，尤其涉及一种数据查询方法与设备。
背景技术
在大数据平台查询数据时需要提供数据查询条件，数据查询条件用于通过一个或多个索引字段筛选所需的数据。按照数据查询条件对应的索引字段的数量，数据查询场景可分为点查、范围查询和多维查询等场景。其中，点查是指按照单一索引字段的查询条件进行数据查询；范围查询是指按照少量(例如小于或等于3)索引字段的查询条件进行数据查询；多维查询是指按照多个(例如大于3)索引字段的查询条件进行数据查询。目前，对于不同的数据查询场景的时延要求不同。例如，点查要求毫秒级时延，范围查询要求亚秒级(即小于1秒)时延，多维查询要求秒级时延。
为了满足不同数据查询场景的时延要求，目前针对不同的数据查询场景采用了不同的大数据技术，这也意味着对于不同的数据查询场景需要按照多种大数据技术来分别部署资源，资源消耗较大。
发明内容
本申请实施例提供一种数据查询方法与设备，用于减少资源开销。
第一方面，提供一种数据查询方法，该方法可以由第二节点执行，该第二节点用于执行数据查询业务，第二节点也可以称为数据查询装置。第二节点可以从数据库中查询所需的数据，该数据库中包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。第二节点接收第一请求，该第一请求用于请求获取数据库中的数据，第一请求包括查询语句。第二节点根据第一数据表的描述信息确定至少一个数据块的信息，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块，第一数据表和数据查询条件是根据查询语句确定的，该描述信息用于指示第一数据表对应的数据块。第二节点根据至少一个数据块的信息，从数据库中读取该至少一个数据块。
通过该实施方式，第二节点可以根据第一数据表的描述信息，从第一数据表对应的数据块中确定符合数据查询条件的至少一个数据块的信息。该至少一个数据块是符合数据查询条件的数据块，也就是说至少一个数据块包括了第一请求所需获取的数据，那么读取至少一个数据块就可以得到第一请求所请求的数据。一方面，因为数据库中的多个数据表原本用于数据分析业务，本申请实施例在用于数据分析业务的数据表的基础上实现了数据查询业务，数据分析业务通常根据大量维度的查询条件查询数据并对查询的数据进行数据分析，因而数据分析业务也可以称为多维数据分析或多维查询业务；而数据查询业务用于从数据库中获取数据，数据查询业务通常根据少量维度的查询条件进行数据查询，例如数据查询业务可以包括点查、范围查询或多维查询等业务。可见，本申请实施例可以使得原本用于数据分析业务的数据表能够用于点查、范围查询或多维查询等场景，使得多种数据查询场景可共用同一种大数据查询技术的数据表，从而不同的数据查询场景不必使用各自的大数据查询技术，因此也不必按照多种大数据查询技术分别部署资源，而是对于共用的一种大数据查询技术部署资源即可，有利于减少资源开销。另一方面，第一数据表的描述信息能够指示第一数据表对应的数据块，例如指示数据块所包括的数据，因而第二节点可结合该描述信息以及数据查询条件确定第一数据表中满足数据查询条件的数据块，数据查询过程的复杂度低，提升了数据查询性能。
在一种可能的实施方式中，第二节点可根据是否存储描述信息自适应的选择确定符合数据查询条件的至少一个数据块的方式。在第二节点存储了描述信息的情况下，第二节点可以根据存储的描述信息，确定至少一个数据块的信息。通过该方式，第二节点可以在本地确定符合数据查询条件的至少一个数据块，可节省请求其他节点确定数据查询条件的至少一个数据块的通信开销，有助于提升数据查询效率。或者，在第二节点未存储描述信息的情况下，第二节点可以向第一节点发送第二请求，并接收来自第一节点的第一响应。其中，第一节点用于存储数据库中数据表的描述信息，并根据描述信息确定符合数据查询条件的至少一个数据块的信息。第二请求包括第一数据表的标识和数据查询条件，第一响应包括至少一个数据块的信息。通过该方式，第二节点无需存储描述信息，降低了第二节点的存储资源占用。
在一种可能的实施方式中，至少一个数据块的信息包括如下的一项或多项。至少一个数据块分别所属的表文件的标识、至少一个数据块分别在所属的表文件中的地址偏移；或，至少一个数据块的长度。通过这些信息可以确定至少一个数据块在数据库中的存储位置，从而成功的从数据库中读取至少一个数据块。
在一种可能的实施方式中，在第一数据表的类型为第一类型时，第二节点从第一节点获得并存储描述信息。其中，第一类型的数据表对应的数据的数据量小于或等于第一阈值，因此从第一类型的数据表对应的数据块中确定满足数据查询条件的数据块所需的时间较少，对于第二节点中其他功能的影响也是较小的，第二节点存储第一类型的数据表的描述信息之后，则可以由第二节点可以在本地确定符合数据查询条件的数据块，可节省请求其他节点确定数据查询条件的通信开销，有助于提升数据查询效率。
在一种可能的实施方式中，第二节点可接收来自第一节点的第一时长的信息，并且在第一时长小于或等于第二阈值时，确定第一数据表的类型为第一类型。第一时长为第一节点确定至少一个数据块所消耗的时长，第一时长小于或等于第二阈值，也就是从第一类型的数据表对应的数据块中确定满足数据查询条件时所需的时间较少，则可以反映出第一数据表对应的数据的数据量较少，那么就可以确定第一数据表的类型为第一类型。
在一种可能的实施方式中，第二阈值是根据第一往返时延确定的。第一往返时延为第二请求的发送时间与第一响应的接收时间之间的时长。根据第一往返时延确定第二阈值，可以衡量出第一时长与第一往返时延之间的大小关系，例如当第一时长更加接近于第二往返时延时，表明第一往返时延中大部分的时间都是用于确定满足数据查询条件的数据块的，第二节点与第一节点之间的数据传输时长仅占据小部分的时间，那么请求第一节点执行确定过程可以使得第二节点的资源消耗节省得更多。相反的，当第一时长更加接近于第二往返时延时，表明第一往返时延中大部分的时间都是用于第二节点与第一节点之间的数据传输，那么请求第一节点执行确定过程则使得第二节点的资源消耗节省得更少。因此，通过根据第一往返时延设置第二阈值，那么第二节点确定是否将第一数据表确定为第一类型的数据表时，实质上将第二节点所节省的资源消耗的多少作为了考虑因素，有利于平衡第二节点的资源消耗与通信开销。
在一种可能的实施方式中，描述信息包括如下的一项或多项。元数据，元数据用于指示第一数据表包括的第一索引字段，其中，第一数据表包括多个分区，多个分区是根据第一索引字段的值确定的。第一指示信息，用于指示多个分区分别对应的表文件，其中多个分区中的第一分区对应的表文件包括第一分区对应的数据，多个分区对应的其中一个表文件包括一个或多个数据块，其中的一个数据块包括一种或多种索引字段的值。第二指示信息，用于指示多个分区对应的其中一个表文件包括的数据块的索引字段的取值范围。通过这些信息，第二节点可以准确地从第一数据表对应的数据块中确定出满足数据查询条件的至少一个数据块。
在一种可能的实施方式中，数据查询条件包括至少一种索引字段的值。那么第二节点根据元数据确定至少一种索引字段包括第一索引字段之后，可以从多个分区中确定第二分区，第二分区是至少一个索引字段中的第一索引字段的值所在的分区。进而，第二节点可根据第一指示信息，确定第二分区对应的表文件，最后根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。通过该实施方式，第二节点可以根据用于划分分区的第一索引字段筛除不符合数据查询条件的分区，减少所需查询的分区的数量，从而提升数据查询效率。
在一种可能的实施方式中，数据查询条件包括至少一种索引字段的值。那么第二节点根据元数据确定至少一种索引字段未包括第一索引字段之后，根据第一指示信息，确定第一数据表对应的表文件，最后根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。通过该实施方式，第二节点确定数据查询条件中的索引字段不包括第一索引字段时，会对第一数据表对应的所有表文件的数据块进行查询，从中确定出满足数据查询条件的至少一个数据块，从而将第一数据表中不符合数据查询条件的数据块筛除，减少了所需读取的数据块的数量，提升了数据查询性能。
第二方面，本申请实施例还提供了一种数据查询方法，该方法可以由第一节点执行。第一节点接收来自第二节点的第二请求，第二请求包括第一数据表的标识和数据查询条件。第一数据表包括在数据库包括的多个数据表中，该数据库多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。第一节点根据存储的第一数据表的描述信息，确定第一数据表对应的数据块中符合数据查询条件的至少一个数据块，描述信息用于指示第一数据表对应的数据块。第一节点向第二节点发送第一响应，第一响应包括第一指示信息，第一指示信息用于指示至少一个数据块。
通过该实施方式，第一节点可以根据第一数据表的描述信息，从第一数据表对应的数据块中确定符合数据查询条件的至少一个数据块的信息。一方面，因为数据库中的多个数据表原本用于数据分析业务，本申请实施例在用于数据分析业务的数据表的基础上实现了数据查询业务，可见，本申请实施例可以使得原本用于数据分析业务的数据表可以用于点查、范围查询或多维查询等场景，实现了多种数据查询场景可共用一种数据库技术的数据表，就无需像相关技术中需要对于不同的数据查询场景分别部署用于存储多种数据库技术对应的数据表的存储资源，从而本申请实施例降低了数据表存储所需的资源消耗。另一方面，第一数据表的描述信息能够指示第一数据表对应的数据块，例如指示数据块所包括的数据，因而可结合该描述信息以及数据查询条件确定第一数据表中满足数据查询条件的数据块，数据查询过程的复杂度低，提升了数据查询性能。
在一种可能的实施方式中，第一节点用于存储数据库中的数据表的描述信息。第一节点可通过多种方式获得描述信息。一种方式为，第一节点响应于预加载指示，读取数据库中多个数据表对应的表文件，并根据多个数据表对应的表文件，确定多个数据表的描述信息，通过该方式，第一节点可预加载所有数据表的描述信息。另一种方式为，第一节点根据来自第三节点的第三请求，读取第二数据表对应的表文件，并根据第二数据表对应的表文件，确定第二数据表的描述信息，第三请求用于指示第三节点对第二数据表进行更新，第三节点用于维护数据库，通过该方式，第一节点可以在数据库中数据更新时及时更新描述信息，提升描述信息的准确性，降低数据查询失败的概率。
在一种可能的实施方式中，至少一个数据块的信息包括如下一项或多项：至少一个数据块分别所属的表文件的标识；至少一个数据块分别在所属的表文件中的地址偏移；或，至少一个数据块的长度。
在一种可能的实施方式中，向第二节点发送第一时长的信息，第一时长为第一节点确定至少一个数据块所消耗的时长。
在一种可能的实施方式中，描述信息包括如下一项或多项。元数据，元数据用于指示第一数据表包括的第一索引字段，其中，第一数据表包括多个分区，多个分区是根据第一索引字段的值确定的。第一指示信息，用于指示多个分区分别对应的表文件，其中多个分区中的第一分区对应的表文件包括第一分区对应的数据，多个分区对应的其中一个表文件包括一个或多个数据块，其中的一个数据块包括一种或多种索引字段的值。第二指示信息，用于指示多个分区对应的其中一个表文件包括的数据块的索引字段的取值范围。
在一种可能的实施方式中，数据查询条件包括至少一种索引字段的值。第一节点根据元数据确定至少一种索引字段包括第一索引字段之后，可从多个分区中确定第二分区，第二分区是至少一个索引字段中的第一索引字段的值所在的分区，并根据第一指示信息，确定第二分区对应的表文件，最后根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
在一种可能的实施方式中，数据查询条件包括至少一种索引字段的值。第一节点根据元数据确定至少一种索引字段未包括第一索引字段之后，可根据第一指示信息，确定第一数据表对应的表文件，最后根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
第三方面，本申请实施例还提供了一种数据查询装置，该数据查询装置具有实现上述第一方面的方法实施例中的功能，有益效果可以参见第一方面的描述，此处不再赘述。功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。
在一个可能的设计中，该数据查询装置包括接收模块、确定模块以及数据读取模块。这些模块可以执行上述第一方面方法示例中的相应功能，具体参见方法示例中的详细描述，此处不做赘述。
例如，接收模块用于接收第一请求，第一请求用于请求获取数据库中的数据，第一请求包括查询语句；其中，数据库包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。确定模块用于根据第一数据表的描述信息确定至少一个数据块的信息，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块，第一数据表和数据查询条件是根据查询语句确定的，描述信息用于指示第一数据表对应的数据块。数据读取模块用于根据至少一个数据块的信息，从数据库中读取至少一个数据块。
通过上述装置，可以根据第一数据表的描述信息，从第一数据表对应的数据块中确定符合数据查询条件的至少一个数据块的信息。该至少一个数据块是符合数据查询条件的数据块，也就是说至少一个数据块包括了第一请求所需获取的数据，那么读取至少一个数据块就可以得到第一请求所请求的数据。一方面，因为数据库中的多个数据表原本用于数据分析业务，本申请实施例在用于数据分析业务的数据表的基础上实现了数据查询业务，数据分析业务通常根据大量维度的查询条件查询数据并对查询的数据进行数据分析，因而数据分析业务也可以称为多维数据分析或多维查询业务；而数据查询业务用于从数据库中获取数据，数据查询业务通常根据少量维度的查询条件进行数据查询，例如数据查询业务可以是点查、范围查询或多维查询业务等。可见，本申请实施例可以使得原本用于数据分析业务的数据表可以用于点查、范围查询或多维查询等场景，实现了多种数据查询场景可共用一种数据库技术的数据表，就无需像相关技术中需要对于不同的数据查询场景分别部署用于存储多种数据库技术对应的数据表的存储资源，从而本申请实施例降低了数据表存储所需的资源消耗。另一方面，第一数据表的描述信息能够指示第一数据表对应的数据块，例如指示数据块所包括的数据，因而可结合该描述信息以及数据查询条件确定第一数据表中满足数据查询条件的数据块，数据查询过程的复杂度低，提升了数据查询性能。
第四方面，本申请实施例还提供了一种数据查询装置，该数据查询装置具有实现上述第二方面的方法实施例中的功能，有益效果可以参见第二方面的描述，此处不再赘述。功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。
在一个可能的设计中，该数据查询装置包括接收模块、确定模块以及发送模块。这些模块可以执行上述第二方面方法示例中的相应功能，具体参见方法示例中的详细描述，此处不做赘述。
例如，接收模块用于接收来自第二节点的第二请求，第二请求包括第一数据表的标识和数据查询条件；其中，第一数据表包括在数据库中，数据库系统中的数据库包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。确定模块用于根据存储的第一数据表的描述信息，确定第一数据表对应的数据块中符合数据查询条件的至少一个数据块，描述信息用于指示第一数据表对应的数据块。发送模块用于向第二节点发送第一响应，第一响应包括第一指示信息，第一指示信息用于指示至少一个数据块。
通过上述装置，数据查询装置可以根据第一数据表的描述信息，从第一数据表对应的数据块中确定符合数据查询条件的至少一个数据块的信息。一方面，因为数据库中的多个数据表原本用于数据分析业务，本申请实施例在用于数据分析业务的数据表的基础上实现了数据查询业务，可见，本申请实施例可以使得原本用于数据分析业务的数据表可以用于点查、范围查询或多维查询等场景，实现了多种数据查询场景可共用一种数据库技术的数据表，就无需像相关技术中需要对于不同的数据查询场景分别部署用于存储多种数据库技术对应的数据表的存储资源，从而本申请实施例降低了数据表存储所需的资源消耗。另一方面，第一数据表的描述信息能够指示第一数据表对应的数据块，例如指示数据块所包括的数据，因而可结合该描述信息以及数据查询条件确定第一数据表中满足数据查询条件的数据块，数据查询过程的复杂度低，提升了数据查询性能。
第五方面，本申请还提供了一种大数据处理系统，包括上述第三方面和第四方面的数据查询装置。可选的，该大数据处理系统还包括数据分析节点，用于执行数据分析业务，以及对数据库进行维护。
第六方面，本申请还提供了一种计算机设备，计算机设备包括处理器和存储器，还可以包括通信接口，处理器执行存储器中的程序指令执行上述第一方面或第一方面任一可能的实现方式提供的方法。或者处理器执行存储器中的程序指令执行上述第二方面或第二方面任一可能的实现方式提供的方法。存储器与处理器耦合，其保存数据查询过程中必要的计算机程序指令和数据。通信接口，用于与其他设备进行通信，如接收第一请求、第二请求、第一响应，或者发送第二请求、第一响应等。
第七方面，本申请还提供了一种计算机设备，计算机设备包括加速装置、处理器，可选的，还可以包括存储器、通信接口，处理器与加速装置配合执行上述第一方面或第一方面任一可能的实现方式提供的方法。或加速装置执行上述第一方面或第一方面任一可能的实现方式提供的方法。存储器与处理器耦合，其保存数据查询过程中需要的一些计算机程序指令和数据。通信接口，用于与其他设备进行通信，如接收第一请求、第二请求、第一响应等。或者，处理器与加速装置配合执行上述第二方面或第二方面任一可能的实现方式提供的方法。或加速装置执行上述第二方面或第二方面任一可能的实现方式提供的方法。通信接口，用于与其他设备进行通信，如发送第二请求、第一响应等。
第八方面，本申请提供了一种计算机系统，该计算机系统包括至少一个计算机设备。每个计算机设备包括存储器和处理器。至少一个计算机设备的处理器用于访问存储器中的代码以执行第一方面或第一方面的任意一种可能的实现方式提供的方法，或者至少一个计算设备的处理器用于访问存储器中的代码以执行第二方面或第二方面的任意一种可能的实现方式提供的方法。
第九方面，本申请提供了一种计算机可读存储介质，计算机可读存储介质被计算机设备执行时，计算机设备执行前述第一方面或第一方面的任意可能的实现方式中提供的方法或执行前述第二方面或第二方面的任意可能的实现方式中提供的方法。该存储介质中存储了计算机程序指令。该存储介质包括但不限于易失性存储器，例如随机访问存储器，非易失性存储器，例如快闪存储器、硬盘(hard disk drive，HDD)、固态硬盘(solid state drive，SSD)。
第十方面，本申请提供了一种计算机程序产品，计算机程序产品包括计算机程序指令，在计算机程序指令被计算机设备执行时，计算机设备执行前述第一方面或第一方面的任意可能的实现方式中提供的方法或执行前述第二方面或第二方面的任意可能的实现方式中提供的方法。该计算机程序产品可以为一个软件安装包，在需要使用前述第一方面或第一方面的任意可能的实现方式中提供的方法或需要使用前述第二方面或第二方面的任意可能的实现方式中提供的方法的情况下，可以下载该计算机程序产品并在计算设备上执行该计算机程序产品。
第十一方面，本申请还提供一种计算机芯片或芯片系统，该芯片或芯片系统与存储器相连，该芯片或芯片系统用于读取并执行存储器中存储的计算机程序指令，执行上述第一方面以及第一方面的各个可能的实现方式中的方法，或者执行上述第二方面以及第二方面的各个可能的实现方式中的方法。
上述第三方面至第十一方面及其实现方式的有益效果可以参考对第一方面或第二方面及其任一项实现方式的有益效果的描述。
附图说明
图1为本申请实施例提供的方案的一种应用场景示意图；
图2A和图2B为相关技术中数据查询过程的流程示意图；
图3为申请实施例提供的数据库系统的系统架构示意图；
图4为本申请实施例提供的数据查询方法的一种流程示意图；
图5为本申请实施例提供的数据查询方法的另一种流程示意图；
图6为本申请实施例提供的第一节点获取描述信息的流程示意图；
图7为本申请实施例提供的数据查询方法的又一种流程示意图；
图8为本申请实施例提供的第二节点存储第一数据表的描述信息的流程示意图；
图9为本申请实施例提供的确定至少一个数据块的信息的过程的流程示意图；
图10为本申请实施例提供的一种确定数据块的过程的示意图；
图11为本申请实施例提供的一种数据查询装置的一种结构示意图；
图12为本申请实施例提供的另一种数据查询装置的一种结构示意图；
图13为本申请实施例提供的数据查询装置的另一种结构示意图。
具体实施方式
为了使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施例作进一步地详细描述。
本申请实施例中，对于名词的数目，除非特别说明，表示“单数名词或复数名词”，即"一个或多个”。“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B的情况，其中A,B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。例如，A/B，表示：A或B。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a,b,或c中的至少一项(个)，表示：a,b,c,a和b,a和c,b和c,或a和b和c，其中a,b,c可以是单个，也可以是多个。
本申请实施例提及“第一”、“第二”等序数词是用于对多个对象进行区分，不用于限定多个对象的大小、内容、顺序、时序、优先级或者重要程度等。例如，第一数据块和第三数据块，可以是同一个数据块，也可以是不同的数据块。另外，本申请所介绍的各个实施例中对于步骤的编号，只是为了区分不同的步骤，并不用于限定步骤之间的先后顺序。
在介绍本申请实施例提供的具体方案之前，对本申请中的部分用语或概念进行解释说明，以便于本领域技术人员理解。
(1)、数据库(database)、数据表(table)、表文件(file)、数据块(data block)
数据库是按照一定的数据结构来组织、存储和管理数据的仓库，数据库是按照特定的格式把数据存储起来，用户可以对存储的数据进行增、删、改、查操作。在某种意义上，数据库是一个文件集合，数据库本质上也可以理解成一个文件系统。在一个示例中，数据库是指大数据平台中的数据仓库，用于存储大数据平台中涉及到的数据。
数据表是数据库中用来存储数据的对象。数据表也可以理解为有结构的数据的集合，也是一种结构化数据的组织形式，通常以行(rows)和列(columns)的形式存储数据。每一行代表表中的一条记录(record)，每一列则代表记录中的一个索引字段(field)或属性(attribute)的值。例如，一个用户信息表可能包括索引字段如下：用户身份标识(identity，ID)、姓名、年龄或地址等，每一行则是一个用户的具体信息。
表文件是数据库中用来存储数据表中数据的物理文件，也可以称为数据文件。在数据库中，文件通常指的是文件系统中的物理文件，用来存储数据库的各种数据结构和内容，如表数据、索引、日志等。每个数据库通常由多个文件组成，每个文件负责存储特定类型或特定数据表的数据，例如文件可以包括数据文件(存储表数据)、日志文件(记录数据库操作的日志信息)、索引文件(存储数据库索引的信息)等。在数据库中，通常会把每个数据表存储为一个文件或一组相关文件，这些文件包含了数据表中所有记录的实际数据。在数据库系统中通过表文件来读取和写入数据，以实现数据的持久化存储和高效访问。表文件的物理存储格式和具体实现方式取决于具体的业务场景，不同的业务场景可能有不同的文件组织结构和存储策略。例如，在基于HBase(一种键-值查询和存储引擎)的大数据技术栈中，数据表为HBase格式表，表文件为HFile文件，HBase和HFile格式的结合可以实现快速的点查和范围查询。又例如，在基于Spark(一种多维查询和存储引擎)的大数据技术栈中，数据表为Parquet格式表，表文件为Parquet文件，Spark和Parquet格式的结合可以实现高效的多维数据分析。
数据块是表文件中的一段数据。例如，一个数据块可以是一个Parquet文件中的一个行分组(RowGroup)。
(2)查询语句、索引字段
查询语句是用来从数据库中检索数据的命令或指令。它指定了需要返回哪些数据以及如何对数据进行筛选、排序和组合。常见的查询命令例如为SELECT(用于选择要从数据库中检索的数据)命令。此外，查询命令还包括命令的子句，例如FROM和WHERE是用于构建查询语句的两个关键子句，FROM用于指定要查询的表或视图的标识，WHERE用于筛选满足特定条件的数据行。
索引字段，也可以称为数据表中数据的字段或属性。索引字段是数据表中的一个或多个列，通常会通过创建索引结构来加快查询操作的速度。例如前述的示例中，一个用户信息表可能包括索引字段如下：用户ID、姓名、年龄或地址等。索引结构可以理解为一种数据结构，类似于书籍的目录，可以快速定位到特定内容，常见的索引数据结构包括B树(B-tree)、哈希表(Hash)、全文索引等，每种索引结构适用于不同的查询模式和数据特性。
(3)多维数据分析
多维数据分析也可以称为在线分析处理(on-line analysis processing，OLAP)，是一种用于探索和分析复杂数据的技术和方法。它的核心概念是将数据从多个角度进行分析，以便深入理解数据的关系、趋势和模式。多维数据模型是多维数据分析的基础，它由多个维度(Dimensions)和度量(Measures)组成。维度是数据的描述性属性，一种索引字段可以作为一个维度，度量是需要分析的数值数据。维度是数据分析的主要依据，通常是描述性的属性或分类，例如时间、地理位置、产品类别等。维度的不同组合形成了多维数据空间。度量是在多维数据分析中被分析的数值数据，例如销售额、数量、利润等。在多维数据分析中，数据通常被组织为立方体的形式，也称为数据立方体或OLAP立方体。
(4)数据跳读(data skipping)
数据查询的过程包括查询计划生成(query plan generation)和查询执行(query execution)，查询计划(query plan)是指在执行SQL查询时生成的详细步骤和策略，用于描述如何检索所需的数据，通常包括访问路径、连接方法、过滤条件应用顺序以及排序方式等内容。数据跳读是一种对查询计划进行优化技术。执行数据跳读时，可以基于索引字段的值，过滤掉不满足数据查询条件的文件，这个优化过程也被称为“剪枝”，可细分为分区剪枝和数据剪枝，分区剪枝是裁剪掉数据表的分区中不满足查询条件的分区，数据剪枝是裁剪掉分区内不满足查询条件的文件。在本申请实施例中，数据跳读可直接过滤掉不满足数据查询条件的数据块，相较于相关技术中数据跳读而言数据粒度更细，能够有效的提升数据查询的速度。
请参见图1，图1示出了本申请实施例的方法可适用的一种场景示意图。在该场景中可包括服务器101和终端设备102。在一个示例中，参见图1所示，该服务器101可以为大数据平台的服务器，大数据平台可包括一个或多个计算机设备1011组成的计算集群。用户可通过终端设备102向服务器101发送相应的业务请求，以使用大数据平台提供的服务，例如数据查询业务请求或数据分析业务请求等。
大数据平台也可以称为大数据处理系统或大数据系统等，能够有效地处理、存储和分析海量数据。大数据平台的核心是大数据技术栈，大数据技术栈是指用于处理和分析大规模数据的一系列技术和工具的集合，这些工具能够有效地收集、存储、处理和分析大量的数据，以从中提取有价值的信息。参见图1所示，如下是大数据技术栈中常见的一些关键组成部分：
(1)数据采集与预处理服务：这一阶段涉及从各种来源收集数据，并进行必要的清洗和转换，以确保数据的质量和适用性。
(2)数据存储服务：大数据平台需要能够高效地存储和管理海量数据，并支持多源异构数据的存储，在一个示例中，大数据平台的数据存储可以通过Hadoop分布式文件系统(hadoop distributed file system，HDFS)等来实现。
(3)数据处理与计算服务：大数据平台可利用各种计算框架对数据进行深入分析，以及支持复杂的查询和数据处理任务。在一个示例中，大数据平台的数据处理与计算可通过Apache Spark或Hadoop MapReduce等实现，其中，Apache Spark是一种快速通用的大数据处理引擎，支持批处理、交互式查询和流处理，或者Hadoop MapReduce是用于大规模数据集的并行分布式计算模型。
(4)数据查询与分析服务：该服务的目的是从海量数据中提取有用信息并进行分析。较为常见的大数据查询与分析工具包括Apache Spark和Apache HBase等，Spark是一个快速通用的大数据处理引擎，支持内存计算，提供了丰富的应用程序接口(application program interface，API)，可以用于实时数据处理和复杂分析，HBase是一个分布式的、面向列的NoSQL数据库，适合存储大量结构化数据，并支持实时查询和分析。
(5)数据挖掘与应用服务：通过机器学习、深度学习等技术，从数据中挖掘出潜在的关联、模式和趋势，为业务决策提供支持。
此外，大数据平台还需要考虑安全性、可扩展性、高性能等方面，以确保系统的稳定运行和数据的可靠保护。大数据技术栈的选择取决于具体的业务需求、数据规模、实时性要求以及可扩展性需求。企业通常会根据自身的情况组合和集成这些技术和工具，以构建适合其大数据处理需求的解决方案。随着技术的发展，大数据系统正变得越来越全流程、可视化和智能化，为企业级应用提供了强大的支持。
终端设备102可以是个人使用的计算机设备，例如，智能手机、笔记本电脑或台式电脑等，也可以是用户侧的服务器，对此不做限制。服务器101包括的各个计算机设备1011可以是服务器，例如，本地服务器、云服务器或数据中心中的服务器。
在一个示例中，服务器101可以为数据库系统的服务器，那么服务器101可向终端设备102提供数据库相关业务服务。
在一些示例中，目前针对不同的数据查询场景采用了不同的大数据技术，这也意味着需要同时满足多种数据查询场景的时延要求时，需要按照多种大数据技术的需求来部署设备资源，资源消耗量大。其中，不同的大数据技术基于各自私有的文件格式，提供不同的查询引擎的能力。例如，如下示例了两种不同的数据库技术方案。
第一种：采用键值(key/value，K/V)存储和检索技术来实现的方案
该方案采用键值对来存储数据，每个键都会对应一个唯一的值，很适合将主键(primary key)作为查询条件来进行数据查询，但是将非主键作为查询条件时，需要进行全表扫描，数据查询时延很高。主键的值能唯一地标识数据表中的每一行，这样的一列或多列数据称为数据表的主键。
请参见图2A，图2A为采用K/V存储和检索技术实现数据查询过程的一种流程示意图。在该示例中，以HBase引擎与HFile数据格式结合实现数据查询为例，HBase引擎是一种常见的K/V查询引擎，HFile数据格式是其私有的数据格式，两者结合可以实现快速的按Key点查或范围查询。该方案的实现过程包括如下几个阶段：
S10：HBase表创建阶段。例如当存在用户的数据需要存储时，且不存在相应的数据表时，HBase引擎创建HBase表。其中，每一个Hbase表中按key进行全局排序，一个Hbase表可横向切分为多个小表，每个小表对应了多个HFile文件。
S11：数据加载阶段。在数据加载阶段中，Hbase服务器中的HBase引擎需要将数据按照HFile数据格式存储为HFile文件，以方便后续进行数据查询。
为了提升加载性能，参见图2A所示，HBase引擎借助于Spark引擎来批量生成HFile文件。在用户存在数据存入或更新时，即图2A所示的加载请求，Spark引擎来生成HFile文件，并将生成的HFile文件发送给HBase引擎，HBase引擎将HBase引擎发送给HDFS进行存储，存储完成之后向客户端返回相应的响应结果。其中，每个HFile文件内的数据按key排序，并分成了多个block(例如每个block的大小为64K)，HFile文件中记录了block的索引，每个block中以K/V数据格式存储数据，也就是说，针对每个value必须存储一个key。
S12：数据查询阶段。当需要查询数据时，则HBase引擎会根据用户的查询请求，从HDFS获取数据。在该实现方案中，通过元数据(Meta)表保存每个小表和存储它的服务器之间的对应关系。
参见图2A所示，首次查询时客户端未缓存Meta表，则需要请求HBase引擎获取并缓存Meta表。在按Key检索时，客户端先访问Meta表确定出该Key所在的小表，然后获取小表所在的Hbase服务器，进而向该Hbase服务器发送点查/范围查询请求，以请求Hbase服务器获取所需的数据，Hbase服务器中的Hbase引擎读取并解析HFile文件，利用block索引构建B-Tree索引结构，然后根据B-Tree索引结构定位出所需的数据所在的block。最后扫描Block中的数据，返回包括该数据块中满足条件的K/V的查询结果给客户端。
在采用K/V存储技术的数据库中，按key查询(即查询条件为主键)时能够满足点查或范围查询的时延要求，因此该方案可适用于点查或范围查询场景，但由于多维查询时往往涉及到非key查询(即非主键作为查询条件)，而非key查询需要进行整个数据库中的数据表的扫描，时延较高，无法满足多维查询的时延要求。并且，使用K/V数据结构存储关系型数据表的数据时会导致存储空间线性膨胀，占用存储资源较多。
第二种：采用在线分析处理(on-line analysis processing，OLAP)技术来实现的方案
OLAP是一种用于分析和查询大规模数据集的数据处理技术。OLAP通过多维数据模型进行分析处理，多维数据模型由多个维度和度量组成。适合OLAP技术的数据格式例如包括Parquet数据格式，Parquet数据格式是一种列式存储格式。Spark引擎是一种多维查询引擎，Spark引擎与Parquet数据格式的数据表结合可以实现高效的OLAP过程，也能实现点查或范围查询。请参见图2B，图2B为Spark引擎与Parquet数据格式结合实现点查或范围查询过程的一种流程示意图。该流程包括如下阶段：
S20：Parquet数据格式的数据表(简称为Parquet表)创建阶段。在表创建阶段，Spark引擎创建Parquet表，且为了加速点查或范围查询，Spark引擎对每个数据表划分分区/分桶，即创建分区/分桶的Parquet表。Parquet表对应的表文件称为Parquet文件，一个分区/分桶可以对应多个Parquet文件。
S21：数据加载阶段。在数据加载阶段中，在用户存在数据存入或更新时，即图2B所示的加载请求，Spark引擎执行数据加载过程，将数据按key(如高频查询key)写入到对应的分区和/或分桶，在每个分桶内数据按key排序，最终生成相应的Parquet文件，并将Parquet文件存储在HDFS中。
S22：数据查询阶段。Spark采用分布式集群架构，包括Driver进程和Executor进程，Driver进程和Executor进程可以部署在不同的设备节点中。在数据查询过程中，需要通过该架构中的成员配合完成整个数据查询过程。数据查询过程包括如下步骤：
S221：Driver进程生成查询计划。在生成查询计划时，Driver进程会从HDFS读取索引文件，根据该索引文件确定满足查询条件的Parquet文件，因而最终生成的查询计划包括了所需查询的Parquet文件列表，相当于生成查询计划的过程即为一次粗选满足查询条件的Parquet文件的过程，然后该查询计划执行时会被划分为多个子任务，每个子任务处理Parquet文件列表中的部分Parquet文件，多个子任务会被分配给多个Executor进程来执行。
S222：Executor进程分别执行自己的子任务，得到满足查询条件的数据行。参见图2B所示，一个Executor进程执行子任务时，该Executor进程会根据分配给自己的子任务中的Parquet文件进行查询。由于Driver进程只会提供Parquet文件列表，而不会提供任何索引信息，因此Executor进程还会从HDFS中读取Parquet文件列表中各Parquet文件的页脚(footer)，来获取元数据，进而根据元数据和查询条件，获取满足查询条件的数据块。之后，Executor进程读取数据块，从读取的数据块中过滤出满足查询条件的数据行。
最后，Driver进程收集将收集Executor进程执行子任务得到的数据行，然后返回给客户端。
由于Spark引擎的设计初衷是为了实现多维数据分析，因此Spark引擎提供了庞大的分布式任务协同框架以及复杂的查询流程，来满足多维数据分析过程的计算需求，但这会导致按Key点查/范围查询时的时延较高，通常时延在秒级，不满足点查或范围查询的时延要求。
根据上述的两种方案可以看到，目前均是基于各种大叔技术私有的数据格式，来提供不同的查询引擎的能力。例如上述的第一种方案无法识别Parquet数据格式，并且不支持自定义数据源，无法兼容多维查询的需求，而上述第二种方案的数据查询流程过于繁琐，使得点查或范围查询时的查询效率较低。因此，若是想要兼容满足点查或者范围查询的时延要求和满足多维查询的时延要求，需要按照多种大数据技术的需求来分别部署设备资源，例如数据库需要同时存储两种不同数据格式的文件，这显然会占用过多的设备存储资源，资源消耗较大。
基于此，本申请实施例提供了一种数据查询方法，该方法在接收第一请求时，可以根据第一数据表的描述信息，从第一数据表对应的数据块中确定符合数据查询条件的至少一个数据块的信息。该至少一个数据块是符合数据查询条件的数据块，也就是说至少一个数据块包括了第一请求所需获取的数据，那么读取至少一个数据块就可以得到第一请求所请求的数据。一方面，本申请实施例可以使得原本用于数据分析业务的数据表可以用于点查、范围查询或多维查询等场景，实现了多种数据查询场景可共用一种数据库技术的数据表，就无需像相关技术中需要对于不同的数据查询场景分别部署用于存储多种数据库技术对应的数据表的存储资源，从而本申请实施例降低了数据表存储所需的资源消耗。另一方面，第一数据表的描述信息能够指示第一数据表对应的数据块，例如指示数据块所包括的数据，因而可结合该描述信息以及数据查询条件确定第一数据表中满足数据查询条件的数据块，数据查询过程的复杂度低，提升了数据查询性能。例如，相较于上述第二种数据库技术，本申请实施例在基于数据分析业务的数据表的基础上，可有效的提升点查或范围查询等数据查询业务的数据查询性能。
本申请实施例提供的方法可以应用于大数据平台或数据库系统中。这里以大数据平台为例，图3为本申请实施例提供的大数据平台的系统架构示意图。其中，在该系统架构中包括数据查询业务节点301、数据分析业务节点302、投影服务节点303和数据库304。
数据查询业务节点301用于处理数据查询业务。数据查询业务用于从数据库304中查询所需的数据，该数据被存储在该数据库304中。如图3所示，客户端可向数据查询业务节点301发送数据查询请求，数据查询业务节点301执行数据查询过程获得相应的数据，并向该客户端返回响应，在该响应中仅包括从数据库304中查询到的数据。例如，数据查询业务可以包括点查或范围查询业务。
在一些示例中，数据查询业务节点301包括数据查询引擎，数据查询引擎可以称为K/V Server，K/V Server是本申请实施例提供的一个处理数据查询业务(如点查/范围查询)的轻量级查询引擎。
数据分析业务节点302用于处理数据分析业务。数据分析业务用于根据数据库304中的数据进行数据分析，因此执行数据分析业务时，实质上也需要先从数据库304中获取相应的数据，然后根据获取的数据进行数据分析，那么最终返回给客户端的响应中会包含数据分析的结果。例如，数据分析业务可以包括多维数据分析(或OLAP)业务。
投影服务节点303用于执行数据跳读(data skipping)。投影服务节点303中存储了数据库中各个数据表的描述信息，该描述信息用于指示数据库中各个数据表对应的数据块中的数据。描述信息也可以称为索引信息。投影服务节点303可通过预加载过程，预先加载并存储数据表的描述信息，并在数据表中数据变更时增量更新描述信息。如图3所示，数据查询业务节点301对接收的数据查询请求进行解析后，可包括生成查询计划和执行查询计划两个过程，在生成查询计划时，则可以请求投影服务节点303执行数据跳读过程。在执行数据跳读时，投影服务节点303可根据自身存储的需查询的数据表的描述信息，从需查询的数据表对应的数据块中确定满足数据查询条件的数据块，返回这些数据块的信息给数据查询业务节点301。数据查询业务节点301通过这些数据块的信息精准定位并读取数据库304中的数据块，解析这些数据块中各满足数据查询条件的记录给客户端。这样，就可以减少数据查询业务节点301执行查询计划时所需读取和查询的数据块的数量，以提升数据查询效率。
数据库304用于存储数据以及元数据，该数据库304为大数据平台的数据仓库，数据库304的数据可以由一个或者多个存储节点来存储。数据库304存储了各个数据表对应的表文件，因此数据库本质上也可以理解成一个文件系统。在一些示例中，数据库304可以采用分布式存储架构，即数据库304可以为分布式文件系统。数据库304可包括一个或多个数据表，这些数据表可用于实现数据查询业务和数据分析业务的基础。例如，数据库304中的数据表可用于进行点查、范围查询和多维查询场景中，因此，对于数据库中的一份数据而言，可支持实现多种不同查询场景的业务，那么无需像相关技术中若想兼容不同查询场景的时延要求就需要同时存储两种不同数据格式的文件，从而本申请实施例的方案可有效的降低数据存储资源的消耗。本申请实施例并不限定数据库304的具体类型。该数据库304能够在数据查询业务节点301、数据分析业务节点302或其他节点的指示下存储数据或元数据，还可以在数据查询业务节点301、数据分析业务节点302的指示下向其传输数据，例如当数据查询业务节点301在查询到数据块后，可以根据数据块的地址从数据库304获取该数据块，响应的数据库304会向数据查询业务节点301传输该数据块。该数据库304可以为具备存储功能的计算机设备或计算机设备集群，该数据库304可以包括计算机设备中具备存储功能的组件，如硬盘、磁盘等。凡是具备数据存储功能的设备或装置均适用于本申请实施例。
如图3所示，数据表可按照分区、分桶的方式进行布局，以存储在数据库304中，用于分区的字段可称为分区字段，分区字段的值可以是数据表中某个索引字段的可能值，例如时间字段的值，一个时段可以划分为一个分区；或者，也可以是数据表中某个索引字段的可能值经过转换后的值，例如数据表中包含的一个索引字段为用户的手机号，那么分区字段的值可以是手机号的最后一个数字，每个数字可以对应一个分区。一个数据表对应的一个或多个表文件中，每个表文件可以按照排序字段来排序数据，每个表文件包括的数据块同样也可以按照排序字段排序数据，并对所有索引字段记录Min-Max统计信息，还可对除排序字段之外的其他字段创建其它类型的索引。排序字段是指用于对数据表中数据进行排序的索引字段，例如可以是数据表中的高频索引字段，高频索引字段是指查询频率最高的索引字段，高频索引字段也可以称为高频查询key。其他字段例如为次高频索引字段，次高频索引字段是指查询频率次高的索引字段。Min-Max统计信息是指一个索引字段的值中的最小值和最大值。
在一些示例中，该系统架构中的各个节点的数量可以为一个或多个。在各个节点的数量为多个的情况下，多个节点可以采用分布式架构。例如，多个数据查询业务节点301可以采用分布式架构，那么客户端发送的数据查询请求可以根据地理位置、负载均衡等策略被分到适当的数据查询业务节点301上，以提升数据查询请求的响应速度。又例如，多个投影服务节点303可以采用分布式架构，实现对描述信息进行分布式缓存，例如多个投影服务节点303可以基于一致性哈希(consistent hashing)算法实现描述信息的分布式缓存，或者也可以采用其他的算法来保证各个投影服务节点303之间的数据一致性。
在一些示例中，本申请实施例并不限定数据查询业务节点301、数据分析业务节点302、投影服务节点303的具体形态，这些节点可以为硬件装置，例如可以为单个计算机设备，也可以为包括多个计算机设备的集群。或者，这些节点可以为虚拟机。再或者，这些节点可以为计算设备中的某个硬件组件，如该计算设备中的处理器(如中央处理器(central processing unit，CPU)、数据处理单元(data processing unit，DPU))、卸载卡、加速卡等。亦或者，这些节点也可以为软件装置，例如可以为用于管理文件的软件，如查询业务软件、数据分析业务软件、数据跳读软件等。数据查询业务节点301、数据分析业务节点302、投影服务节点303可部署在不同的计算机设备中，也可以部署在相同的计算机设备中，本申请实施例对此不做限制。例如，数据查询业务节点301和数据分析业务节点302部署在相同的计算机设备中。
本申请实施例中并不限定存储装置200以及查询装置100的部署方式。该存储装置200与查询装置100可以部署在同一个计算设备中；例如，该查询装置100可以为计算设备中的处理器(如CPU、DPU)，该存储装置200为计算设备中的硬盘。该存储装置200与查询装置100可以部署在不同的计算设备中；例如，该存储装置200与查询装置100可以部署存储系统中，该查询装置100为该存储系统中的计算节点(具备数据计算功能，承担该存储系统中计算任务)，该存储装置200为该存储系统中的存储节点(具备数据存储功能)。
下面结合图1的场景和图3的系统架构来对本申请实施例提供的数据查询方法进行说明。
本申请实施例中，该数据查询方法中涉及到的数据存储在数据库中，数据库中的数据可以采用适用于数据分析业务的数据格式，例如parquet数据格式或其他任意可能的数据格式。在采用该数据查询方法进行数据查询之前，需要在数据库中创建数据表，并执行数据加载过程。例如采用parquet数据格式时，可以创建parquet表，并通过数据加载过程生成parquet表文件，并将parquet表文件存储在数据库中。其中，由于数据库中的表文件是适用于数据分析业务的数据格式，因此，这些表文件理所应当的可应用于数据分析业务，同时，本申请实施例的数据查询方法提供一种复杂度较低的查询流程，来在这些表文件的基础上，提升数据查询的效率，提升数据查询性能，这也意味着通过该数据查询方法可以实现“一份数据”支持多种查询场景的使用的效果，那么也就无须将相关技术中部署不同数据格式来满足不同查询场景的需求，有助于降低存储资源的占用。
请参见图4，图4为本申请实施例提供的数据查询方法的流程示意图。该方法可以由第二节点执行，该第二节点例如可以为图3所示的数据查询业务节点301或数据分析业务节点302。
步骤401：第二节点(K/V server)接收第一请求，第一请求用于请求获取数据库中的数据，第一请求包括查询语句；其中，该数据库包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。
在一些示例中，第一请求可以为用户在客户端进行操作触发的请求，客户端可以是指具有该数据库的访问权限的任意客户端，该客户端可以是指软件客户端，也可以是软件客户端中的小程序等，或者也可以是指浏览器中的网页。例如，用户在使用客户端时输入了用于筛选数据的相关条件，并进行确认操作，客户端就会生成相应的查询语句，并向第二节点发送第一请求。
在一些示例中，第一请求还可以是用户在使用客户端时自动触发的。例如用户使用客户端打开某个页面时，该页面需要显示数据库中的数据，那么客户端就会向第二节点发送第一请求。或者第一请求也可以为其他情况时触发的，本申请实施例不限制第一请求的触发方式。
第一请求和第二节点的类型可以是相对应的。在一个示例中，第一请求可以是数据查询业务对应的请求，相应的，第二节点可以是数据查询业务节点。例如，第一请求可以为用于点查或者范围查询的请求。在另一个示例中，第一请求还可以是数据分析业务对应的请求，相应的，第二节点可以是数据分析业务节点。例如，第一请求可以是用于OLAP分析的请求。在一些场景中，数据查询业务和数据分析业务作为两种不同的业务，可以对应不同的业务页面，当客户端中打开的页面为数据查询业务的页面时，客户端向数据查询业务节点发送第一请求，当客户端中打开的页面为数据分析业务的页面时，客户端向数据分析业务节点发送第一请求。
在第一请求为数据查询业务对应的请求时，由于执行数据分析业务(如OLAP分析)的引擎(如Spark引擎)采用的通用分布式查询架构过于复杂，因此本申请实施例在第二节点中提供了用于数据查询的引擎，该引擎可以称为K/V server，K/V Server是本申请实施例提供的一个处理数据查询业务(如点查/范围查询)的轻量级查询引擎，旨在表明通过本申请实施例的方案可以使得点查或范围查询能够像相关技术中基于K/V技术的方案一样满足点查或范围查询的时延要求，图4中具体是以此为例进行示出的。数据查询业务包括点查或范围查询时，第一请求包括的查询语句所包括的数据查询条件包括的索引字段的种类通常较少，例如小于或等于N种，N的值可以为3或其他值。
在一个示例中，该数据库可以为大数据平台的数据仓库。数据库中包括多个数据表，数据表是数据库中用来存储数据的对象。数据表也可以理解为有结构的数据的集合，也是一种结构化数据的组织形式，通常以行和列的形式存储数据。每一行代表表中的一条记录，每一列则代表记录中的一个索引字段的值。第一请求用于指示根据数据查询条件从数据表中获取一个或多个记录，数据查询条件通常包括数据表中一个或多个索引字段的值。
在本申请实施例中，多个数据表用于数据分析业务，换句话说，多个数据表采用了适合数据分析业务的数据格式。例如，数据表的数据格式可以为parquet格式，那么数据表也可以称为parquet格式表或parquet表。parquet格式是一种高效的列式存储文件格式，通常用于大数据处理和分析场景中，如上述相关技术中2中的方案，通过Spark引擎和parquet格式表的结合可以实现高效的OLAP分析，相较于基于K/V存储技术的数据表，parquet格式表占据的存储空间更少，但是parquet格式表的缺点在于用于点查或范围查询时的性能无法满足点查或范围查询的时延要求。但通过本申请实施例的方案，将在使用parquet格式表的基础上，提升点查或范围查询等数据查询业务的查询效率，从而提升满足点查或范围查询的时延要求的概率，具体方案将在后续进行介绍。也就是说，本申请实施例的方案复用了原本用于数据分析业务的数据，来实现了对数据查询效率的提升，这样一来，对于数据库中存储的每一份数据，即可以用于数据分析业务，也可以用于数据查询业务(如点查或范围查询)，实现了一份数据支持多种查询场景，或者说多种查询场景可使用同一种大数据技术的数据表的效果，从而有效地节省了数据存储所需的资源消耗。
在一些示例中，第一请求可以采用JS对象简谱(java script object notation，JSON)格式描述查询语句。或者，也可以采用其他格式进行描述，本申请实施例对此不做限制。
步骤402：第二节点(K/V server)根据第一数据表的描述信息确定至少一个数据块的信息，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块，第一数据表是根据第一请求确定的，描述信息用于指示第一数据表对应的数据块。
第一请求包括查询语句，通过对查询语句进行解析，可确定需要查询的数据表和数据查询条件，第一数据表即为根据查询语句确定的数据表。也就是说，在接收第一请求之后，可以对第一请求中的查询语句进行解析，以确定需要查询的第一数据表。查询语句包括了至少一种查询命令和子句，查询命令也可以称为查询函数，用于指定需要返回哪些数据以及如何对数据进行筛选、排序和组合。例如，数据查询条件中指定了一个或多个数据表，那么该一个或多个数据表为第一数据表。或者，数据查询条件未指定数据表，那么数据库中的所有数据表为第一数据表。
描述信息用于指示数据库中各个数据表对应的数据块中的数据，那么可以根据该描述信息，确定第一数据表对应的数据块中符合数据查询条件的数据块，以减少需读取的数据块的数量。描述信息也可以称为索引信息或索引。在一个示例中，一个数据表对应一个描述信息，该描述信息用于指示该数据表对应的数据块中的数据。一个数据表的数据被存储在一个或多个表文件，其中的一个表文件可以包括一个或多个数据块，一个数据块可包括一个或多个索引字段的值，索引字段的值实质上即为数据表中的数据。一个数据表对应的数据块即为该数据表对应的表文件包括的数据块。
在一个示例中，描述信息用于指示数据块中的数据的取值范围，也可以描述为指示数据块包括的索引字段的值的取值范围，通过该取值范围可以辅助过滤需查询的数据不在的数据块，以减少需读取的数据块的数量。
步骤403：第二节点(K/V server)根据至少一个数据块的信息，从数据库中读取至少一个数据块。
至少一个数据块包括第一请求请求获取的数据。至少一个数据块的信息中每个数据块的信息可包括该数据块所属的表文件的标识、该数据块在所属的文件中的地址偏移(offset)或该数据块的长度(length)中的一项或多项。其中，表文件的标识可以为该表文件的名称或ID等。这样，通过至少一个数据块的信息，就可以从数据库中读取至少一个数据块。
在一个示例中，数据库可以通过HDFS来实现，在后续的介绍中均以HDFS为例，那么可根据至少一个数据块的信息，从HDFS中读取至少一个数据块。例如，对于至少一个数据块中的每一个数据块，在读取数据块时，第二节点可以调用HDFS的seek接口和read接口，调用seek接口时以offset作为参数，调用read接口时以length作为参数。相应的，HDFS会将该offset和length指定的数据块发送给第二节点。HDFS通过二进制流的方式发送数据块，第二节点可以结合该数据块的元数据将二进制流构造成为一个数据块。
而在如图2B所示的相关技术中，Spark需要首先在Driver进程生成查询计划，在生成查询计划时，Driver进程会从HDFS读取索引文件，根据该索引文件确定满足查询条件的Parquet文件，因而最终生成的而该查询计划只是包括了所需查询的Parquet文件列表，相当于生成查询计划的过程即为一次粗选满足查询条件的Parquet文件的过程，然后该查询计划执行时会被划分为多个子任务，每个子任务处理Parquet文件列表中的部分Parquet文件，多个子任务会被分配给多个Executor进程来执行。每个Executor进程分别执行自己的子任务，得到满足查询条件的数据行，一个Executor进程执行子任务时，该Executor进程会根据分配给自己的子任务中的Parquet文件进行查询。由于Driver进程只会提供Parquet文件列表，而不会提供任何索引信息，因此Executor进程还会从HDFS中读取Parquet文件列表中各Parquet文件的页脚(footer)，来获取元数据，进而根据元数据和查询条件，获取满足查询条件的数据块。这相当于需要与HDFS交互两次，筛除过程也需要执行两次，这是因为Spark的通用分布式架构所决定的，因而用于点查或者范围查询时的查询效率极低，时延较高。
而本申请实施例的方案通过预先加载描述信息，以预先存储分散于表文件中的索引信息，且该描述信息能够指示各表文件的数据块，从而根据该描述信息就可以直接定位满足数据查询条件的数据块，而无需像上述相关技术中需要多次交互和数据筛除过程，可见本申请实施例的方案的查询过程的复杂度较低，从而查询效率更高，数据查询时延更低。
步骤404：第二节点(K/V server)过滤至少一个数据块中满足数据查询条件的记录，并将满足数据查询条件的记录发送给客户端。
对于至少一个数据块中的每一个数据块，可以根据数据查询条件筛选其中满足条件的记录，一个记录即为第一数据表中的一行。之后，将满足数据查询条件的记录包括的数据发送给客户端。其中，返回给客户端的数据可以包括各个记录包括的全部数据，也可以包括各个记录包括的部分索引字段的数据。
通过上述方法，可以根据第一数据表的描述信息，从第一数据表对应的数据块中确定符合数据查询条件的至少一个数据块的信息。该至少一个数据块是符合数据查询条件的数据块，也就是说至少一个数据块包括了第一请求所需获取的数据，那么读取至少一个数据块就可以得到第一请求所请求的数据。一方面，本申请实施例可以使得原本用于数据分析业务的数据表能够用于点查、范围查询或多维查询等场景，使得多种数据查询场景可共用同一种数据库技术的数据表，从而不同的数据查询场景不必使用各自的数据库技术，因此也不必按照多种数据库技术分别部署资源，而是对于共用的一种数据库技术部署资源即可，有利于减少资源开销。另一方面，第一数据表的描述信息能够指示第一数据表对应的数据块，例如指示数据块所包括的数据，因而第二节点可结合该描述信息以及数据查询条件确定第一数据表中满足数据查询条件的数据块，数据查询过程的复杂度低，提升了数据查询性能。
请参见图5所示，为本申请实施例提供的数据查询方法的另一种流程示意图。与图4所示的流程的区别在于第二节点在接收第一请求之后，需要对第一请求中的查询语句进行解析，解析出需要查询的第一数据表、数据查询条件、需要查询的索引字段。需要查询的第一数据表是指查询数据的范围，在查询语句中指定第一数据表时，则只会对该第一数据表进行查询。数据查询条件也可以称为数据筛选条件，用于筛选第一数据表满足特定条件的数据行。需要查询的索引字段用于指示需要返回给客户端的数据对应的索引字段。该数据查询方法包括如下步骤：
步骤501：第二节点(K/V server)接收第一请求，第一请求用于请求获取数据库中的数据，第一请求包括查询语句。
步骤502：第二节点(K/V server)解析第一请求中的查询语句，确定第一数据表和数据查询条件。
在一个示例中，查询语句包括如下内容的一项或多项：
(1)FROM子句，用于指定要查询的数据表的标识。该标识可以为数据表的ID或名称。第一数据表为FROM子句指定的数据表。FROM子句指定的数据表可以包括一个或多个。当指定的数据表包括多个时，第一数据表的数量也是多个，或者说第一数据表包括多个数据表。在多个第一数据表的情况下，对于其中的每个第一数据表都可以采用后续的方式来查询该第一数据表，因此在后续的介绍中，主要是以一个第一数据表为例进行说明的，后续不再进行赘述。例如，数据库中包括的一个数据表的名称为“employees”，那么一条SELECT命令的子句为“FROM employees”，这条命令用于指定从employees表中查询数据。
(2)SELECT命令，用于选择要从数据库中查询的数据。例如，employees表包括以下列：id、name、position和salary，每一列为一种索引字段，那么一条SELECT命令为“SELECT name，salary”，这条命令表示将查询name和salary列的数据，也就是说最终返回给客户端的数据中将包括name和salary列的数据。
(3)WHERE子句，用于筛选满足特定条件的数据行，或者说用于指定选择数据时的条件。它用于过滤记录，只返回数据表中满足特定条件的记录。例如，一条SELECT命令的子句为“WHERE position＝'Manager'”，这条命令表示查询position为Manager的记录。
除了上述的命令之外，查询语句也可以包括其他的命令，对此不做限制。
示例性的，继续沿用上述employees表的示例，第一请求包括的查询语句如下：
“SELECT name，salary FROM employees
WHERE position＝'Manager'”
这条命令表示从employees表中获取position为Manager的员工的name和salary列的数据。
那么，第二节点在接收第一请求之后，则会对第一请求中包括的查询语句进行解析，得到第一请求所要查询的第一数据表、数据查询条件和需要查询的索引字段。
步骤503：第二节点(K/V server)根据第一数据表的描述信息确定至少一个数据块的信息，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块，描述信息用于指示第一数据表对应的数据块。
步骤504：第二节点(K/V server)根据至少一个数据块的信息，从数据库中读取至少一个数据块。
步骤505：第二节点(K/V server)过滤至少一个数据块中满足数据查询条件的记录，并将满足数据查询条件的记录发送给客户端。
在一个示例中，在解析第一请求中的查询语句还可以得到需要查询的索引字段，那么可以将满足数据查询条件的记录中，需要查询的索引字段对应的数据返回给客户端。
上述图5中的部分步骤与图4对应的实施例中的步骤相同，因此对于这些步骤可以参见图4部分的介绍，在此不再进行赘述。
请参见图2B所示的相关技术，在该相关技术中，数据分析节点将表文件的描述信息拉取到查询引擎的Driver进程中进行Data Skipping，但是这将导致数据分析节点的进程内存占用过多，而数据分析节点中还需要内存用于其他的功能，进程内存上限限制，使得数据跳读的过程效率不高，最终导致数据查询效率低。因此，本申请实施例提供了第一节点提供专有的Data Skipping服务，第一节点例如为图3所示的投影服务节点。为了第一节点实现Data Skipping过程，第一节点需要存储数据库包括的多个数据表的描述信息。
请参见图6，图6为第一节点获取描述信息的流程示意图。以数据库为HDFS为例，第一节点获取描述信息可以包括但不限于如下的两种方式：
方式1：第一节点从HDFS预加载数据表的描述信息。
参见图6所示，在该方式中，第一节点可以执行步骤611，即从HDFS中读取表文件，并执行步骤612，根据各个数据表对应的表文件，确定并存储多个数据表的描述信息。例如，在初次使用时，第一节点可以从HDFS读取表文件获取描述信息。又例如，第一节点可以定时从HDFS读取表文件来获取描述信息。
通过该方式，第一节点可以将分散于多维分析文件(即表文件)中的描述信息，预加载到第一节点中，从而便于第一节点后续为第二节点提供Data Skipping服务时，可以根据这些描述信息来实现，提升数据查询效率。
方式2：第一节点可以根据对HDFS的更新，更新存储的描述信息。
在该方式中，大数据平台还包括第三节点，第三节点可用于维护HDFS。例如第三节点可以为图3所示的数据分析业务节点(如Spark引擎节点)。第三节点可以根据对HDFS的操作更新HDFS的数据，当第三节点更新HDFS时，也可以通知第一节点进行描述信息的更新。例如，参见图6所示的步骤621，第三节点可以向第一节点发送第三请求，相应的第一节点接收第三请求，第三请求用于指示第三节点对第二数据表进行更新，那么第一节点执行步骤622，即可以根据该第三请求从HDFS读取第二数据表的表文件，并执行步骤623，根据第二数据表的表文件，确定并存储第二数据表的描述信息。
在一个示例中，可以在第三节点中调用第一节点的部署软件开发工具包(software development kit，SDK)，当第三节点更新HDFS时，就可以调用该SDK向第一节点发送第三请求。
在一个示例中，第三请求可以包括第三节点对HDFS进行的操作以及操作对象，例如插入数据、删除数据、修改数据的操作等。这样，第一节点就可以根据该操作和操作对象确定第二数据表，从而来更新第二数据表的描述信息。
在一个示例中，第一节点确定第二数据表的描述信息，可以是读取完整的第二数据表的表文件，重新确定第二数据表的描述信息。或者，第一节点也可以只读取第二数据表的部分数据、部分表文件或部分数据块，并根据读取的内容更新第二数据表的描述信息，实现对第二数据表的描述信息的增量更新。增量更新的方式能够减少数据传输量，降低第一节点的负载压力，从而有更多的资源用于进行Data Skipping，有助于提升数据查询效率。
在第一节点存储了描述信息之后，就可以根据这些描述信息进行Data Skipping过程了。但是，由于数据库中的不同数据表之间的数据量差别大，针对不同的数据表进行Data Skipping过程所耗费的时长不同。并且，第二节点请求第一节点执行Data Skipping时，第一节点和第二节点之间存在交互开销，若是交互开销占据了网络往返时延(round trip time，RTT)的大部分时间，而Data Skipping过程只是占据了RTT的小部分时间，单独设置第一节点就不再存在优势。
因此，本申请实施例在此基础上提供了一种根据大小表自适应的选择本地Data Skipping或者远程Data Skipping的方案，本地Data Skipping是指由第二节点执行Data Skipping，远程Data Skipping是指第二节点请求第一节点执行Data Skipping过程。大小表主要在于数据表的数据的数据量的大小，当一个数据表的数据量大于第一阈值时该数据表为大表，当一个数据表的数据量小于或等于第一阈值时该数据表为小表。第一阈值的值可根据实际情况进行设置，例如为经验值或实验测定的值。在本申请实施例中，小表称为第一类型的数据表，大表称为第二类型的数据表。
参见图7所示，图7为本申请实施例提供的数据查询方法的又一种流程示意图。
步骤701：客户端向第二节点(K/V server)发送第一请求。第二节点(K/V server)接收第一请求，第一请求用于请求获取数据库中的数据，第一请求包括查询语句。
步骤702：第二节点(K/V server)解析第一请求中的查询语句，确定第一数据表和数据查询条件。
步骤703：第二节点(K/V server)判断是否已存储第一数据表的描述信息。
步骤704：若步骤703的判断结果为是，即在第二节点(K/V server)已存储第一数据表的描述信息的情况下，第二节点(K/V server)根据第一数据表的描述信息确定至少一个数据块的信息。其中，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块。
当第一数据表为第一类型的数据表时，第二节点会存储第一数据表的描述信息，那么第二节点会执行本地Data Skipping过程，即根据自身存储的该描述信息，确定符合数据查询条件的至少一个数据块的信息。第一类型的数据表为上述的小表，此类型的数据表的Data Skipping过程消耗的时长较短，若是请求第一节点执行Data Skipping过程，第二节点和第一节点之间的交互开销占据了RTT的大部分时长，而由第二节点自己执行Data Skipping过程时，由于第一数据表为小表，占用的资源消耗相对较小，并且可以节省与第一节点之间的交互开销，降低第一数据表的Data Skipping过程的时长，从而提升数据查询效率。
在一个示例中，第二节点可以向第一节点发送请求来第一数据表的描述信息。或者，第二节点也可以从数据库读取第一数据表对应的表文件，进而根据读取得到的表文件得到第一数据表的描述信息。
步骤705：若步骤703的判断结果为否，即在第二节点(K/V server)未存储第一数据表的描述信息的情况下，第二节点(K/V server)向第一节点(投影服务节点)发送第二请求。
在本申请实施例中，第二请求包括第一数据表的标识和数据查询条件。
在如下情况时，第二节点未存储第一数据表的描述信息：
情况1：第一数据表为第二类型的数据表，那么第二节点不会存储该类型的数据表的描述信息。第一类型的数据表为上述的大表，此类型的数据表的Data Skipping过程消耗的时长较长，会占用第二节点的较多的设备资源，因此为了避免影响第二节点中其他功能的执行，则可以请求专用的第一节点来执行Data Skipping过程。
情况2：第二节点首次查询第一数据表。在第二节点首次查询第一数据表时，第二节点在此之前不会对第一数据表进行类型的判断，因此第二节点不会存储第一数据表的描述信息。
步骤706：第一节点(投影服务节点)根据存储的第一数据表的描述信息，确定至少一个数据块的信息。
步骤707：第一节点(投影服务节点)向第二节点(K/V server)发送第一响应。相应的，第二节点(K/V server)接收来自第一节点的第一响应。其中，第一响应包括至少一个数据块的信息。
也就是说，当第二节点未存储第一数据表的描述信息时，那么第二节点会执行远程Data Skipping过程，即向第一节点发送第二请求以请求第一节点来执行Data Skipping，第一节点会根据其存储的第一数据表的描述信息，来确定符合数据查询条件的至少一个数据块。
步骤708：第二节点(K/V server)根据至少一个数据块的信息，从数据库中读取至少一个数据块。
步骤709：第二节点(K/V server)过滤至少一个数据块中满足数据查询条件的记录。
步骤710：第二节点(K/V server)将满足数据查询条件的记录发送给客户端。
上述图7中的部分步骤与图4或图5对应的实施例中的步骤相同，因此对于这些步骤可以参见图4或图5对应部分的介绍，在此不再进行赘述。
请参见图8，图8示出了第二节点存储第一数据表的描述信息的流程示意图。
步骤801：第一节点(投影服务节点)向第二节点(K/V server)发送第一时长的信息。第二节点(K/V server)接收来自第一节点(投影服务节点)的第一时长的信息。
第一时长为第一节点确定至少一个数据块所消耗的时长，即第一节点执行Data Skipping过程消耗的时长。
在一个示例中，第一时长的信息可以包括在第一响应中。或者，第一节点还可以在向第二节点发送第一响应之前或之后，向第二节点发送第一时长的信息。
步骤801为可选的步骤，因此在图8中以虚线进行示出。
步骤802：第二节点(K/V server)确定第一数据表的类型为第一类型。
在一个示例中，第二节点可以根据第一数据表对应的数据的数据量确定第一数据表的类型是否为第一类型。如果第一数据表对应的数据的数据量小于或等于第一阈值，第一数据表的类型为第一类型，否则，如果第一数据表对应的数据的数据量大于第一阈值，第一数据表的类型为第二类型。
在一个示例中，第二节点还可以根据第一时长的信息确定第一数据表的类型是否为第一类型。如果第一时长小于或等于第二阈值，确定第一数据表的类型为第一类型，否则，如果第一时长大于第二阈值，确定第一数据表的类型为第二类型。
其中，第二阈值可以为固定值，例如为经验或者实验测定的固定值。或者，第二阈值还可以是根据第一往返时延确定的，例如第二阈值可以为第一往返时延与第一比例的乘积。第一比例的值可以根据需求进行配置。第一往返时延为第二请求的发送时间与第一响应的接收时间之间的时长，该发送时间可以为第二请求的起始发送时刻、结束发送时刻或二者的中值等，该接收时间可以为第一响应的起始接收时刻、结束接收时刻或二者的中值等。例如，第一往返时延为第二节点请求第一节点执行Data Skipping时的RTT。
步骤803：第二节点(K/V server)向第一节点(投影服务节点)发送获取第一数据表的描述信息的请求。
步骤804：第一节点(投影服务节点)向第二节点(K/V server)发送第一数据表的描述信息。
步骤805：第二节点(K/V server)存储描述信息。
下面，对第二节点或第一节点执行Data Skipping过程，即根据第一数据表的描述信息确定至少一个数据块的信息的过程进行介绍。通常而言，Data Skipping过程包括分区(和/或分桶)剪枝和数据剪枝两个过程，分区(和/或分桶)剪枝树查询条件中包含的分区(和/或分桶)条件，从缓存的文件列表中，选择和分区(和/或分桶)条件匹配的结果，数据剪枝基于分区(和/或分桶)剪枝的结果，对其中的每一个表文件，判断其是否满足数据查询条件，如果满足则保留该表文件，否则排除该表文件。
相关技术中，如图2B所示，Spark在Driver进程中生成查询计划，该查询计划提供满足数据查询条件的表文件列表，然后分发给分布式的Executor进程分布式的扫描查询计划提供的文件列表，由于Executor进程不包括描述信息，Executor进程还需要从数据库中读取表文件的页脚等信息，然后再确定满足条件的数据块，相当于描述信息需要加载两次，Data Skipping过程也需要执行两次，这是因为Spark的通用分布式架构所决定的，因而用于点查或者范围查询时的查询效率极低，时延较高。
在本申请实施例中，通过预先存储描述信息，并根据描述信息直接确定数据所在的数据块，则无需多次加载描述信息，并且也无需执行多次Data Skipping过程，可有效提升如点查或范围查询等查询方式的查询效率，降低响应时延。
首先，对第一数据表的描述信息包括的内容进行介绍。
示例性的，对于一个第一数据表，该第一数据表的描述信息可以包括如下一项或多项：
(1)元数据。
元数据也可以称为关键元数据。元数据用于指示第一数据表包括的索引字段集合或索引字段列表。该元数据还指示第一数据表中的第一索引字段。其中，第一数据表包括多个分区，多个分区是根据第一索引字段的值确定的。第一索引字段可以包括分区字段和/或分桶字段。
参见图3所示，数据库中的数据表采用了分区、分桶的方式进行数据布局。一个数据表(如第一数据表)包括一个或多个分区(partition)，数据表的分区可以是按照该数据表中的其中一个索引字段的值或者其转换之后的值确定的，该索引字段的作用用于划分数据表的分区，因而该索引字段可以称为分区字段，数据表的不同分区对应分区字段的不同取值或取值范围。例如，分区字段为时间，数据表的每个分区可以包括一个连续的时段的数据。在一些示例中，分区可以理解为存储区域，一个数据表的不同分区可对应不同的存储区域，一个分区可以对应一个存储区域。一个分区可以理解为数据表的一个子表。
不同的数据表的分区字段可以相同，也可以不同。分区字段可以为数据表中的任意索引字段，或者也可以是对数据表中任意索引字段进行的转换。例如，分区字段可以为指定的数据表中的特定字段，或者，也可以为数据表中的高频索引字段，高频索引字段是指查询频率最高的索引字段，那么，一个数据表可以按照高频索引字段的值(或其转换后的值，如hash值)，将数据表的数据水平拆分到不同的分区(或存储位置)。对于一个数据表而言，高频索引字段可以在经确定之后就不变化，例如，高频索引字段可以是根据经验从数据表的索引字段中确定的，一旦确定将该高频索引字段作为分区字段之后，则该数据表的分区字段不会发生变化。或者，一个数据表的高频索引字段也可以是变化的，随着高频索引字段的变化，数据表的分区也可以发生变化。
一个数据表的一个分区可以包括一个或多个分桶(bucket)，分桶可以是按照该数据表中的其中一个索引字段的值确定的，该索引字段的作用用于划分数据表的分桶，因而该索引字段也可以称为分桶字段，该不同分桶对应分桶字段的不同取值范围。例如，分桶字段为时间，数据表按照时间拆分到不同的分桶，每个分桶包括一个连续的时段的数据，每个分桶可包括一个连续的子时段的数据。在一些示例中，分桶也可以理解为存储区域，一个分区的不同分桶可对应不同的子存储区域，一个分桶以对应一个子存储区域。
分桶字段可以为数据表中的任意索引字段，例如，分区字段可以为指定的数据表中的特定字段，或者，也可以为数据表中的高频索引字段、次高频索引字段或其他字段，次高频索引字段是指查询频率次高的索引字段，次高频索引字段也可以称为次高查询key，那么，按照高频索引字段或次高频索引字段的值(或其转换后的值，如hash值)，将数据表的数据水平拆分到不同的存储位置。如果结合分区使用，则在每个分区中使用分桶字段拆分数据。
(2)第一指示信息，也可以称为分区文件列表或分区文件集合。
第一指示信息用于指示多个分区分别对应的表文件。如上所述，第一数据表包括多个分区，多个分区中的第一分区对应的表文件包括第一分区对应的数据，第一分区可以为多个分区中的任一分区，多个分区对应的其中一个表文件包括一个或多个数据块，其中的一个数据块包括一种或多种索引字段的值。第一指示信息可以指示第一数据表的多个分区中每个分区对应的表文件。
(3)第二指示信息，也可以称为表文件的min-max统计信息，或数据块的min-max统计信息。
第二指示信息用于指示多个分区对应的其中一个表文件包括的数据块的索引字段的取值范围。在一个示例中，取值范围可以通过min-max统计信息来表征，一个数据块的min-max统计信息指示一个数据块中的最小值和最大值，由于数据块中的数据是按照排序字段的值进行排序的，因此该数据块中索引字段的值均是位于最小值和最大值的区间内的。一个数据块可以包括一个或多个索引字段的值，在一个数据块包括多个索引字段的值时，该数据块的min-max统计信息也可以包括多个索引字段的min-max统计信息。排序字段是指用于对数据表中数据进行排序的索引字段，例如可以是数据表中的高频查询key、次高频索引字段等。
例如，第一数据表对应的表文件可以为Parquet文件，一个Parquet文件包含一个或多个数据块(每个数据块的大小为128MB或者128MB以上)，每数据块将多个数据行逐字段进行列式存储。Parquet文件也记录了各数据块中每个字段的min-max统计信息。
(4)布隆过滤器(bloom filter)，布隆过滤器可用于辅助筛除不包括所需查询的数据的表文件或数据块。bloom filter可以是指第一数据表中一个或多个索引字段的bloom filter。
(5)位图(bitmap)，也可用于辅助确定数据块。
或者，描述信息也可以包括其他任意可能的索引方式的信息，对此本申请实施例不做限制。
数据查询条件包括至少一种索引字段的值，下面对确定符合数据查询条件的至少一个数据块的过程进行介绍。请参见图9，图9为确定至少一个数据块的信息的过程的流程示意图。该过程可以由第一节点执行，也可以由第二节点执行。
步骤901：根据元数据确定至少一种索引字段是否包括第一索引字段。第一索引字段可以包括分区字段和/或分桶字段，图9中具体以第一索引字段为分区字段为例进行介绍。
步骤902：如果为步骤901的判断结果为是，即确定至少一种索引字段包括第一索引字段，则从第一数据表的多个分区中确定第二分区。第二分区是至少一个索引字段中的第一索引字段的值所在的分区。
示例性的，分区字段为时间，数据查询条件中时间的值为第一时段，那么可以从第一数据表的多个分区中确定出第一时段所在的分区。
步骤903：根据第一指示信息，确定第二分区对应的表文件。
第一指示信息也可以称为分区文件列表，用于指示第一数据表的各个分区对应的表文件。那么在确定第二分区之后，可以根据第一指示信息，确定第二分区对应的表文件，以缩小数据查询的范围。
步骤904：根据第二指示信息，确定至少一个数据块的信息。至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
第二指示信息可以指示第一数据表对应的表文件所包括的数据块的取值范围，那么根据第二指示信息中第二分区对应的表文件包括的数据块的取值范围，以及数据查询条件中包括的至少一种索引字段的值，就可以确定出满足数据查询条件的至少一个数据块，获得至少一个数据块的信息。
在一个示例中，取值范围为min-max统计信息，那么对于需要查询的每一个数据块，可以根据至少一种索引字段中每一个索引字段的值是否位于min-max的值之间，若没有位于min-max的值之间，则过滤该数据块，否则，若位于min-max的值之间，那么该数据块可能包括所需查询的数据。
示例性的，参见图10所示，图10为本申请实施例提供的一种确定数据块的过程的示意图。
第一请求指示查询的数据表名称为“tab表”，查询条件为“索引字段no＝1and索引字段c1＝‘d’”，其中，索引字段no为分区字段，索引字段c1为分桶字段，索引字段c1的值“d”通过转换，可以确定该值对应的桶号为1。
“tab表”的描述信息中，分区/分桶文件列表表示为"tab:no＝1:1"->"file1,file2"、"tab:no＝1:2"->"file3"、"tab:no＝2:1"->"file4,file5"，即分区1的分桶1对应表文件“file1”和“file2”，分区1的分桶2对应表文件“file3”，分区2的分桶1对应表文件“file4”和“file5”。file1和file2中，索引字段c1的min-max信息分别为[‘a’,‘e’]和[‘x’,‘y’]，即file1中索引字段c1的最小值为a，最大值为e，file2中索引字段c1的最小值为x，最大值为y。file1中包含了两个数据块，即数据块1和数据块2，索引字段c1在两个数据块中的min-max信息分别为[‘a’,‘b’]和[‘c’,‘e’]。
那么，在执行Data Skipping过程时，分区/分桶剪枝结果为"tab:no＝1:1"对应的文件列表"file1,file2"，数据剪枝结果为file1(‘d’>‘a’and‘d’<‘e‘)，file2被排除(因为‘x’>‘d’导致不满足查询条件)。进一步的，继续排查数据块，则只有数据块2满足查询条件c1＝‘d’，因此返回数据块2的信息，例如数据块2所在的表文件file1，数据块2在该表文件中的offset、length等。
在相关技术(例如上述Spark的方案)中，在执行完Data Skipping过程之后，只会返回一个表文件列表，进而需要调用HDFS的seek和read接口，读取文件页脚，通过解析页脚的索引信息，结合客户端的查询条件，再执行一次类似于Data Skipping的过程，才能获取文件中满足查询条件的数据块信息，然后再去读取HDFS中相应的表文件中的目标数据块。
相关技术在Data Skipping之后访问文件时，还需要再次读取文件，并分析其中的目标数据块，过程较为繁琐。而通过本申请实施例的方法，通过一次Data Skipping过程就可以直接确定满足数据查询条件的数据块，而无需再次从数据库读取页脚并确定数据块，整个查询流程的复杂度更低，且由于数据块数量更少，减少了扫描的输入/输出(input/output，I/O)开销。
例如，按照相关技术的方案，在Driver进程给定文件列表(即file1)之后，由Executor进程从HDFS读取文件file1的页脚，根据页脚保存的索引信息，再结合查询条件，分析出只有数据块2满足条件，最后Driver进程再读取数据块2，而本申请实施例可以由K/V server直接确定出数据块2满足数据查询条件，从而可直接读取数据块2。
步骤905：如果为步骤901的判断结果为否，即确定至少一种索引字段未包括第一索引字段，那么根据第一指示信息，确定第一数据表对应的表文件。也就是说当至少一种索引字段未包括分区字段或者分桶字段时，无法过滤分区，那么会对第一数据表的所有分区的表文件包括的数据块进行查询。
步骤906：根据第二指示信息，确定至少一个数据块的信息。至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
与步骤904类似，根据第一数据表的所有分区的表文件包括的数据块的取值范围，以及数据查询条件中包括的至少一种索引字段的值，就可以确定出满足数据查询条件的至少一个数据块，获得至少一个数据块的信息。
综上所述，本申请实施例中，提供了一个轻量级的查询引擎K/V server，该K/V server接收来自客户端的点查/范围查询请求，解析到查询表和查询条件，然后对基于多维数据分析的数据表的表文件进行Data Skipping过程，确定其中符合数据查询条件的数据块，以减少需要读取的数据块的数量，最后读取数据块获取查询结果。考虑到相关技术将索引信息拉取到查询引擎的Driver进程中进行Data Skipping，由于查询引擎还用于实现其他的功能，这样查询引擎所在设备就会面临进程内存上限的问题，本申请实施例中设置了一个专有的Data Skipping服务节点，即第一节点，以缓解设备内存上限的问题，并且考虑到节点间的交互开销，采用了大小表自适应的选择本地或远程Data Skipping的方案，在交互开销和Data Skipping的资源消耗之间进行平衡。
因而，该数据查询方法可以基于数据分析业务(如OLAP分析)的数据执行高效的点查/范围查询，该方法实现了一种数据格式的数据表支持多种数据查询场景(如点查、范围查询或数据分析业务等)，那么只需要一份数据，相较于相关技术中满足多种查询场景的时延要求时需要部署多种数据格式的数据时，该方法可以节省近乎一半的存储资源，降低了数据管理的成本。并且，相较于相关技术中spark进行点查或范围查询的方案中繁琐的查询流程，本申请实施例的数据查询方法的查询流程复杂度更低，且在进行data skipping时返回的结果的粒度更细(相关技术为文件，本申请实施例的方法中为数据块)，查询过程消耗的时间更短，查询效率更高，例如在进行点查或范围查询时的查询性能更优，可接近于相关技术中K/V查询方案的查询性能。
与上述方法实施例基于相同的技术构思，本申请实施例还提供一种数据查询装置，该数据查询装置用于执行上述方法实施例中的数据查询方法的模块。
在一种示例中，如图11所示，数据查询装置110包括接收模块1101、确定模块1102以及数据读取模块1103。具体地，在数据查询装置110中，各模块之间通过通信通路建立连接。
接收模块1101，用于接收第一请求，第一请求用于请求获取数据库中的数据，第一请求包括查询语句；其中，数据库包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。确定模块1102，用于根据第一数据表的描述信息确定至少一个数据块的信息，至少一个数据块是第一数据表对应的数据块中符合数据查询条件的数据块，第一数据表和数据查询条件是根据第一请求确定的，描述信息用于指示第一数据表对应的数据块。数据读取模块1103，用于根据至少一个数据块的信息，从数据库中读取至少一个数据块。
作为一种可能的实施方式，确定模块1102，具体用于根据存储的描述信息，确定至少一个数据块的信息；或者，在未存储描述信息的情况下，向第一节点发送第二请求，并接收来自第一节点的第一响应，其中，第二请求包括第一数据表的标识和数据查询条件，第一响应包括至少一个数据块的信息。
作为一种可能的实施方式，至少一个数据块的信息包括如下一项或多项：至少一个数据块分别所属的表文件的标识；至少一个数据块分别在所属的表文件中的地址偏移；或，至少一个数据块的长度。
作为一种可能的实施方式，确定模块1102，还用于在第一数据表的类型为第一类型时，从第一节点获得并存储描述信息；其中，第一类型的数据表对应的数据的数据量小于或等于第一阈值。
作为一种可能的实施方式，接收模块1101，还用于接收来自第一节点的第一时长的信息，第一时长为第一节点确定至少一个数据块所消耗的时长。确定模块1102，还用于若第一时长小于或等于第二阈值，确定第一数据表的类型为第一类型。
作为一种可能的实施方式，第二阈值是根据第一往返时延确定的，第一往返时延为第二请求的发送时间与第一响应的接收时间之间的时长。
作为一种可能的实施方式，描述信息包括如下一项或多项：元数据，元数据用于指示第一数据表包括的第一索引字段，其中，第一数据表包括多个分区，多个分区是根据第一索引字段的值确定的；第一指示信息，用于指示多个分区分别对应的表文件，其中多个分区中的第一分区对应的表文件包括第一分区对应的数据，多个分区对应的其中一个表文件包括一个或多个数据块，其中的一个数据块包括一种或多种索引字段的值；第二指示信息，用于指示多个分区对应的其中一个表文件包括的数据块的索引字段的取值范围。
作为一种可能的实施方式，数据查询条件包括至少一种索引字段的值。确定模块1102，具体用于：根据元数据确定至少一种索引字段包括第一索引字段；从多个分区中确定第二分区，第二分区是至少一个索引字段中的第一索引字段的值所在的分区；根据第一指示信息，确定第二分区对应的表文件；根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
作为一种可能的实施方式，数据查询条件包括至少一种索引字段的值。确定模块1102，具体用于：根据元数据确定至少一种索引字段未包括第一索引字段；根据第一指示信息，确定第一数据表对应的表文件；根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
作为一种可能的实施方式，确定模块1102，还用于确定所述至少一个数据块中符合所述数据查询条件的数据行。
在一种示例中，如图12所示，数据查询装置120包括接收模块1201、确定模块1202以及发送模块1203。具体地，在数据查询装置120中，各模块之间通过通信通路建立连接。
接收模块1201，用于接收来自第二节点的第二请求，第二请求包括第一数据表的标识和数据查询条件；其中，第一数据表包括在数据库中，数据库包括多个数据表，多个数据表用于数据分析业务，数据分析业务用于从数据库中获取数据并对获取的数据进行分析。确定模块1202，用于根据存储的第一数据表的描述信息，确定第一数据表对应的数据块中符合数据查询条件的至少一个数据块，描述信息用于指示第一数据表对应的数据块。发送模块1203，用于向第二节点发送第一响应，第一响应包括第一指示信息，第一指示信息用于指示至少一个数据块。
作为一种可能的实施方式，数据查询装置120还包括描述信息加载模块1204。描述信息加载模块1204用于：响应于预加载指示，读取数据库中多个数据表对应的表文件，并根据多个数据表对应的表文件，确定多个数据表的描述信息；或者，根据来自第三节点的第三请求，读取第二数据表对应的表文件，并根据第二数据表对应的表文件，确定第二数据表的描述信息，第三请求用于指示第三节点对第二数据表进行更新，第三节点用于维护数据库。
作为一种可能的实施方式，至少一个数据块的信息包括如下一项或多项：至少一个数据块分别所属的表文件的标识；至少一个数据块分别在所属的表文件中的地址偏移；或，至少一个数据块的长度。
作为一种可能的实施方式，发送模块1203还用于向第二节点发送第一时长的信息，第一时长为第一节点确定至少一个数据块所消耗的时长。
作为一种可能的实施方式，描述信息包括如下一项或多项。元数据，元数据用于指示第一数据表包括的第一索引字段，其中，第一数据表包括多个分区，多个分区是根据第一索引字段的值确定的。第一指示信息，用于指示多个分区分别对应的表文件，其中多个分区中的第一分区对应的表文件包括第一分区对应的数据，多个分区对应的其中一个表文件包括一个或多个数据块，其中的一个数据块包括一种或多种索引字段的值。第二指示信息，用于指示多个分区对应的其中一个表文件包括的数据块的索引字段的取值范围。
作为一种可能的实施方式，数据查询条件包括至少一种索引字段的值。确定模块1202，具体用于：根据元数据确定至少一种索引字段包括第一索引字段，从多个分区中确定第二分区，第二分区是至少一个索引字段中的第一索引字段的值所在的分区，并根据第一指示信息，确定第二分区对应的表文件，最后根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
作为一种可能的实施方式，数据查询条件包括至少一种索引字段的值。确定模块1202，具体用于：根据元数据确定至少一种索引字段未包括第一索引字段；根据第一指示信息，确定第一数据表对应的表文件，最后根据第二指示信息，确定至少一个数据块的信息，至少一个数据块包括至少一种索引字段中一种或多种索引字段的值。
本申请实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，另外，在本申请各个实施例中的各功能模块可以集成在一个处理器中，也可以是单独物理存在，也可以两个或两个以上模块集成为一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。其中，图11和图12的示意中，虚线表示的单元为可选的功能单元。
该集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，手机，或者网络设备等)或处理器(processor)执行本申请各个实施例该方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
在另一些实施例中，本申请实施例提供的数据查询装置130的结构可以如图13所示，包括处理器1301以及与处理器1301连接的存储器1302。处理器1301和存储器1302之间可以通过总线相互连接，处理器1301可以作为数据查询装置130的主处理器，即数据查询装置130的控制核心。总线可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。本申请实施例中不限定上述处理器1301以及存储器1302之间的具体连接介质。
存储器1302可以包括上述的内存，存储器1302可以是易失性存储器，例如随机存取存储器；存储器也可以是非易失性存储器，例如只读存储器ROM，快闪存储器，硬盘(hard disk drive，HDD)或固态硬盘(solid-state drive，SSD)、或者存储器1302是能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器1302可以是上述存储器的组合。存储器1302可用于存储软件程序以及模块。
处理器1301可以包括一个或多个处理器，处理器1301通过运行存储在存储器1302中的软件程序以及模块，从而执行数据查询装置130的各种功能应用以及数据处理，如本申请实施例提供的数据查询方法。
可以理解的是，本申请实施例示意的结构并不构成对数据查询装置的具体限定。在本申请另一些实施例中，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
本申请的实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行计算机程序或指令的方式来实现。计算机程序或指令可以构成计算机程序产品。本申请实施例还提供一种计算机程序产品，包含有计算机可执行指令。在一种实施例中，该计算机可执行指令用于使计算机执行上述方法实施例中的功能。
计算机可执行指令可以被存放于计算机可读存储介质中，本申请实施例还提供一种计算机可读存储介质，所述计算机可读存储介质内存储有可执行指令。在一种实施例中，该计算机可执行指令用于使计算机执行上述方法实施例中的功能。
本申请实施例提供的计算机可读存储介质可以是随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmableROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically ePROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的计算机可读存储介质。
计算机可执行指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘。
本申请实施例中还提供一种计算机程序产品，包括指令，当其在计算机上运行时，使得计算机执行上述第一节点或第二节点所执行的方法。
本申请实施例提供了一种芯片系统，包括处理器和接口，所述处理器用于从所述接口调用并运行指令，使得所述芯片系统实现上述各实施例中的方法。该芯片系统可以由芯片构成，也可以包含芯片和其他分立器件。
本申请实施例还提供一种芯片，包括处理器，用于支持该通信装置实现上述方法实施例中第一节点或第二节点所涉及的功能。在一种可能的设计中，该芯片与存储器连接或者该芯片包括存储器，该存储器用于保存该通信装置必要的计算机程序或指令和数据。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各种说明性逻辑块(illustrative logical block)和步骤(step)，能够以电子硬件、计算机软件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。

Claims (17)

  1. 一种数据查询方法，其特征在于，方法包括：
    接收第一请求，所述第一请求用于请求获取数据库中的数据，所述第一请求包括查询语句；其中，所述数据库包括多个数据表，所述多个数据表用于数据分析业务，所述数据分析业务用于从所述数据库中获取数据并对获取的数据进行分析；
    根据第一数据表的描述信息确定至少一个数据块的信息，所述至少一个数据块是所述第一数据表对应的数据块中符合数据查询条件的数据块，所述第一数据表和所述数据查询条件是根据所述查询语句确定的，所述描述信息用于指示所述第一数据表对应的数据块；
    根据所述至少一个数据块的信息，从所述数据库中读取所述至少一个数据块。
  2. 根据权利要求1所述的方法，其特征在于，所述根据第一数据表的描述信息确定至少一个数据块的信息，包括：
    根据存储的所述描述信息，确定至少一个数据块的信息；或者，
    在未存储所述描述信息的情况下，向第一节点发送第二请求，并接收来自所述第一节点的第一响应，其中，所述第二请求包括所述第一数据表的标识和所述数据查询条件，所述第一响应包括所述至少一个数据块的信息。
  3. 根据权利要求2所述的方法，其特征在于，所述至少一个数据块的信息包括如下一项或多项：
    所述至少一个数据块分别所属的表文件的标识；
    所述至少一个数据块分别在所属的表文件中的地址偏移；或，
    所述至少一个数据块的长度。
  4. 根据权利要求2或3所述的方法，其特征在于，所述方法还包括：
    在所述第一数据表的类型为第一类型时，从所述第一节点获得并存储所述描述信息；其中，所述第一类型的数据表对应的数据的数据量小于或等于第一阈值。
  5. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    接收来自所述第一节点的第一时长的信息，所述第一时长为所述第一节点确定所述至少一个数据块所消耗的时长；
    若所述第一时长小于或等于第二阈值，确定所述第一数据表的类型为所述第一类型。
  6. 根据权利要求5所述的方法，其特征在于，所述第二阈值是根据第一往返时延确定的，所述第一往返时延为所述第二请求的发送时间与所述第一响应的接收时间之间的时长。
  7. 根据权利要求1～6任一所述的方法，其特征在于，所述描述信息包括如下一项或多项：
    元数据，所述元数据用于指示所述第一数据表包括的第一索引字段，其中，所述第一数据表包括多个分区，所述多个分区是根据所述第一索引字段的值确定的；
    第一指示信息，用于指示所述多个分区分别对应的表文件，其中所述多个分区中的第一分区对应的表文件包括所述第一分区对应的数据，所述多个分区对应的其中一个表文件包括一个或多个数据块，其中的一个数据块包括一种或多种索引字段的值；
    第二指示信息，用于指示所述多个分区对应的其中一个表文件包括的数据块的索引字段的取值范围。
  8. 根据权利要求7所述的方法，其特征在于，所述数据查询条件包括至少一种索引字段的值；
    所述根据第一数据表的描述信息确定至少一个数据块的信息，包括：
    根据所述元数据确定所述至少一种索引字段包括第一索引字段；
    从所述多个分区中确定第二分区，所述第二分区是所述至少一个索引字段中的第一索引字段的值所在的分区；
    根据所述第一指示信息，确定所述第二分区对应的表文件；
    根据所述第二指示信息，确定所述至少一个数据块的信息，所述至少一个数据块包括所述至少一种索引字段中一种或多种索引字段的值。
  9. 根据权利要求7所述的方法，其特征在于，所述数据查询条件包括至少一种索引字段的值；
    所述根据第一数据表的描述信息确定至少一个数据块的信息，包括：
    根据所述元数据确定所述至少一种索引字段未包括所述第一索引字段；
    根据所述第一指示信息，确定所述第一数据表对应的表文件；
    根据所述第二指示信息，确定所述至少一个数据块的信息，所述至少一个数据块包括所述至少一种索引字段中一种或多种索引字段的值。
  10. 根据权利要求1～9任一所述的方法，其特征在于，在根据所述至少一个数据块的信息，从所述数据库中读取搜索至少一个数据块之后，所述方法还包括：
    确定所述至少一个数据块中符合所述数据查询条件的数据行。
  11. 一种数据查询方法，其特征在于，所述方法包括：
    接收来自第二节点的第二请求，所述第二请求包括第一数据表的标识和数据查询条件；其中，第一数据表包括在数据库中，所述数据库包括多个数据表，所述多个数据表用于数据分析业务，所述数据分析业务用于从所述数据库中获取数据并对获取的数据进行分析；
    根据存储的所述第一数据表的描述信息，确定所述第一数据表对应的数据块中符合所述数据查询条件的至少一个数据块，所述描述信息用于指示所述第一数据表对应的数据块；
    向所述第二节点发送第一响应，所述第一响应包括第一指示信息，所述第一指示信息用于指示所述至少一个数据块。
  12. 根据权利要求11所述的方法，其特征在于，所述方法还包括：
    响应于预加载指示，读取所述数据库中所述多个数据表对应的表文件，并根据所述多个数据表对应的表文件，确定所述多个数据表的描述信息；或者，
    根据来自第三节点的第三请求，读取第二数据表对应的表文件，并根据所述第二数据表对应的表文件，确定所述第二数据表的描述信息，所述第三请求用于指示所述第三节点对所述第二数据表进行更新，所述第三节点用于维护所述数据库。
  13. 一种数据查询装置，其特征在于，所述数据查询装置包括处理单元和收发单元，所述处理单元与所述收发单元耦合，以执行如权利要求1～10中任一项所述的方法，或执行如权利要求11～12中任一项所述的方法。
  14. 一种计算机设备，其特征在于，包括处理器和存储器，处理器和存储器相连，存储器存储计算机程序，当存储器中存储的计算机程序被处理器执行时，使得装置执行如上述权利要求1～10或11～12中任一项方法。
  15. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质用于存储计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如权利要求1～10中任一项所述的方法，或使得所述计算机执行如权利要求11～12中任一项所述的方法。
  16. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如权利要求1～10中任一项所述的方法，或使得所述计算机执行如权利要求11～12中任一项所述的方法。
  17. 一种芯片系统，其特征在于，所述芯片系统包括：
    处理器和接口，所述处理器用于从所述接口调用并运行指令，当所述处理器执行所述指令时，实现如权利要求1～10中任一项所述的方法，或实现如权利要求11～12中任一项所述的方法。
PCT/CN2025/082684 2024-06-27 2025-03-14 一种数据查询方法与设备 Pending WO2026001091A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410855954.4A CN121233581A (zh) 2024-06-27 2024-06-27 一种数据查询方法与设备
CN202410855954.4 2024-06-27

Publications (1)

Publication Number Publication Date
WO2026001091A1 true WO2026001091A1 (zh) 2026-01-02

Family

ID=98158774

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/082684 Pending WO2026001091A1 (zh) 2024-06-27 2025-03-14 一种数据查询方法与设备

Country Status (2)

Country Link
CN (1) CN121233581A (zh)
WO (1) WO2026001091A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10318491B1 (en) * 2015-03-31 2019-06-11 EMC IP Holding Company LLC Object metadata query with distributed processing systems
CN114546942A (zh) * 2022-01-28 2022-05-27 苏州浪潮智能科技有限公司 一种数据库数据读取方法、装置、终端及存储介质
CN114706861A (zh) * 2022-06-08 2022-07-05 天津南大通用数据技术股份有限公司 一种在列存储引擎中按列动态分组存储的方法
CN114764406A (zh) * 2021-01-12 2022-07-19 中国联合网络通信集团有限公司 一种数据库查询方法及相关装置
CN116783588A (zh) * 2020-11-13 2023-09-19 谷歌有限责任公司 用于大元数据管理的列技术

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10318491B1 (en) * 2015-03-31 2019-06-11 EMC IP Holding Company LLC Object metadata query with distributed processing systems
CN116783588A (zh) * 2020-11-13 2023-09-19 谷歌有限责任公司 用于大元数据管理的列技术
CN114764406A (zh) * 2021-01-12 2022-07-19 中国联合网络通信集团有限公司 一种数据库查询方法及相关装置
CN114546942A (zh) * 2022-01-28 2022-05-27 苏州浪潮智能科技有限公司 一种数据库数据读取方法、装置、终端及存储介质
CN114706861A (zh) * 2022-06-08 2022-07-05 天津南大通用数据技术股份有限公司 一种在列存储引擎中按列动态分组存储的方法

Also Published As

Publication number Publication date
CN121233581A (zh) 2025-12-30

Similar Documents

Publication Publication Date Title
JP6617117B2 (ja) 半構造データのためのスケーラブルな分析プラットフォーム
US7174345B2 (en) Methods and systems for auto-partitioning of schema objects
US20170083573A1 (en) Multi-query optimization
US8924373B2 (en) Query plans with parameter markers in place of object identifiers
US11256684B1 (en) Applying relational algebraic operations to change result sets of source tables to update a materialized view
CN106294695A (zh) 一种面向实时大数据搜索引擎的实现方法
CN115114374A (zh) 事务执行方法、装置、计算设备及存储介质
US11841848B2 (en) Stored procedures for incremental updates to internal tables for materialized views
CN112231351A (zh) 一种pb级海量数据的实时查询方法和装置
US12169493B2 (en) Workload-aware column inprints
US10776368B1 (en) Deriving cardinality values from approximate quantile summaries
CN111078705A (zh) 基于Spark平台建立数据索引方法及数据查询方法
KR20230096107A (ko) 빅 메타데이터 관리를 위한 열 기반 기법들
WO2025180404A1 (zh) 数据处理方法、电子设备和存储介质
Calçada et al. Evaluation of Couchbase, CouchDB and MongoDB using OSSpal.
WO2024239782A1 (zh) 查询计划构建方法、装置、电子设备和存储介质
Liu et al. Using provenance to efficiently improve metadata searching performance in storage systems
US12111833B1 (en) Query acceleration for disjunctive equijoin conditions
US12169491B1 (en) Dynamic selection of plan interpretation to perform queries
CN118708608A (zh) 处理引擎的选择方法、装置、计算机设备、存储介质
WO2026001091A1 (zh) 一种数据查询方法与设备
Singh NoSQL: A new horizon in big data
CN118035270A (zh) 数据查询方法、装置、软件程序、设备及存储介质
CN113742346A (zh) 资产大数据平台架构优化方法
US12265535B1 (en) Dataset summary metadata providing improved query performance

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25824486

Country of ref document: EP

Kind code of ref document: A1