CN113342856B - Data query method and device - Google Patents
Data query method and device Download PDFInfo
- Publication number
- CN113342856B CN113342856B CN202110717857.5A CN202110717857A CN113342856B CN 113342856 B CN113342856 B CN 113342856B CN 202110717857 A CN202110717857 A CN 202110717857A CN 113342856 B CN113342856 B CN 113342856B
- Authority
- CN
- China
- Prior art keywords
- query
- historical data
- node
- segment
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention provides a data query method and a data query device, wherein a first query request is received, and a query bit number in the first query request is extracted; inquiring storage groups corresponding to the inquiry bit numbers and historical data integrity information on the inquiry nodes in the inquiry starting and ending time based on the inquiry bit numbers; dividing the first query request into a plurality of second query requests according to the storage groups corresponding to the query bit numbers; for each second query request, segmenting the second query request according to the query starting and stopping time to obtain a plurality of query segments; and performing data query on each query node to obtain a query result. In the scheme, the first query request is divided into multiple stages through the query bit number to obtain a query segment corresponding to the third query request, data query is performed on each query node to obtain a query result, query efficiency is improved, integrity of query data is guaranteed, and reliability of the query data is improved.
Description
Technical Field
The invention relates to the technical field of data processing in the process industry, in particular to a data query method and a data query device.
Background
In the process industry, enterprises generate a large amount of historical data in production and management, and the historical data is generally used for tracing and summarizing the past, so that the query of the historical data is particularly important.
In the prior art, there are the following ways to query data:
(1) The upper layer directly initiates a query request to a specified node. Although the data on a given node can be queried directly, the node bears all the pressure of a single query, and in the case of data redundancy, the data of the query may be incomplete. Therefore, the method has the problems of performance and data credibility, and the system resources are unbalanced.
(2) The system provides a query agent, the agent queries the nodes, and analyzes and combines the query results of the nodes. This approach relieves the resource pressure to some extent, but does not improve performance.
(3) And sampling the data according to a preset sampling frequency and a preset time span, storing the data into different data groups, and matching different groups according to the query time span during query. According to the method, the primary data of the data source is subjected to secondary sampling, so that the query timeliness is improved to a certain extent, but the accuracy of the primary data query and the result based on the primary data aggregation query cannot be ensured.
Therefore, in the prior art, the data query mode is low in query efficiency, the queried data is incomplete, the queried data is not credible, and the data query efficiency, the data integrity and the data credibility cannot be guaranteed.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for querying a time-series database, an electronic device, and a storage medium, so as to achieve the purposes of improving query efficiency, ensuring integrity of query data, and improving reliability of query data.
In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:
the first aspect of the embodiment of the invention discloses a data query method, which is suitable for a distributed time sequence database, and comprises the following steps:
receiving a first query request and extracting a query bit number in the first query request;
inquiring the storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring historical data integrity information on the inquiry node within the starting time and the ending time;
dividing the first query request into a plurality of second query requests according to the storage group corresponding to the query bit number, wherein the query bit number contained in each second query request is in the same storage group;
for each second query request, dividing the second query request into a plurality of third query requests according to the query starting and stopping time to obtain query segments corresponding to the third query requests, wherein each query segment corresponds to a time period;
and performing data query on each query node based on the historical data integrity information and the query segment on each query node to obtain a query result.
Optionally, for each second query request, the dividing the second query request into a plurality of third query requests according to the query start-stop time to obtain a query segment corresponding to the third query request includes:
and for each second query request, if the query start-stop time is greater than a preset threshold, dividing the second query request into a plurality of third query requests according to a preset time period value to obtain a query segment corresponding to each query node, wherein each query segment comprises all query bit numbers.
Optionally, the performing, on the basis of the historical data integrity information on each query node and the query segment, a data query on each query node to obtain a query result includes:
for each query node, judging whether the historical data in the query segment on the query node is complete or not based on the historical data integrity information on the query node;
if the historical data in the query segments on all the query nodes are complete, combining the historical data in the query segments on each query node to obtain a query result;
if the historical data in the query segments on the partial query nodes are incomplete, determining new query segments on the partial query nodes based on the query starting and ending time and time periods corresponding to other query segments with complete historical data;
combining the historical data in the other query segments with the historical data in the new query segments on the partial query nodes to obtain a query result;
if the historical data in the query segments on all the query nodes are incomplete, determining a new query segment on each query node based on the query starting and ending time, the time period corresponding to each query segment and a preset division strategy;
and combining the historical data in the new query segment on each query node to obtain a query result.
Optionally, the dividing, for each second query request, the second query request into a plurality of third query requests according to the query start/stop time to obtain a query fragment corresponding to the third query request includes:
and splitting the second query request into a plurality of third query requests according to the query bit number if the query start-stop time is smaller than a preset threshold value for each second query request to obtain a query segment corresponding to each query node, wherein the query segment at least comprises one query bit number.
Optionally, the performing, on the basis of the historical data integrity information on each query node and the query segment, a data query on each query node to obtain a query result includes:
for each query node, judging whether the historical data in the query segment on the query node is complete or not based on the historical data integrity information on the query node;
if the historical data in the query segment on any query node is complete, acquiring the historical data in the query segment as a query result;
if the historical data in the query segments on all the query nodes are incomplete, determining the segments with complete historical data on all the query nodes as new query segments;
and performing deduplication processing on historical data corresponding to all the new query segments, and taking the obtained historical data as a query result.
The second aspect of the embodiment of the present invention discloses a data query device, which is suitable for a distributed time sequence database, and the device includes:
the extraction module is used for receiving a first query request and extracting a query bit number in the first query request;
the first query module is used for querying the storage group corresponding to the query bit number based on the query bit number and querying the historical data integrity information on the query node within the starting time and the ending time;
the first dividing module is used for dividing the first query request into a plurality of second query requests according to the storage group corresponding to the query bit number, and the query bit number contained in each second query request is in the same storage group;
the second dividing module is used for dividing each second query request into a plurality of third query requests according to the query starting and ending time to obtain query segments corresponding to the third query requests, and each query segment corresponds to a time period;
and the second query module is used for performing data query on each query node based on the historical data integrity information and the query segment on each query node to obtain a query result.
Optionally, the second dividing module is specifically configured to:
and for each second query request, if the query start-stop time is greater than a preset threshold, dividing the second query request into a plurality of third query requests according to a preset time period value to obtain a query segment corresponding to each query node, wherein each query segment comprises all query bit numbers.
Optionally, the second query module includes:
a first judging unit, configured to judge, for each query node, whether historical data in the query segment on the query node is complete based on historical data integrity information on the query node, execute a first merging unit if historical data in the query segment on all the query nodes is complete, execute a second merging unit if historical data of part of the query nodes in the query segment is incomplete, and execute a third merging unit if historical data of all the query nodes in the query segment is incomplete;
the first merging unit is configured to merge historical data in the query segment on each query node to obtain a query result;
the second merging unit is configured to determine a new query segment on the partial query node based on the query start-stop time and a time period corresponding to another query segment with complete historical data; merging the historical data in the other query segments and the historical data in the new query segments on the partial query nodes to obtain a query result;
the third merging unit is configured to determine a new query segment on each query node based on the query start-stop time, a time period corresponding to each query segment, and a preset partition policy;
and merging the historical data in the new query segment on each query node to obtain a query result.
Optionally, the second dividing module is further specifically configured to:
and for each second query request, if the query start-stop time is less than a preset threshold value, splitting the second query request into a plurality of third query requests according to the query bit number to obtain a query segment corresponding to each query node, where the query segment at least includes one query bit number.
Optionally, the second query module includes:
a second judging unit, configured to judge, for each query node, whether historical data in the query segment on the query node is complete based on historical data integrity information on the query node, execute a fourth merging unit if the historical data in the query segment on any query node is complete, and execute a fifth merging unit if the historical data in the query segment of all the query nodes is incomplete;
the fourth merging unit is configured to obtain historical data in the query segment as a query result;
the fifth merging unit is used for determining the segments with complete historical data on all the query nodes as new query segments; and performing deduplication processing on the historical data corresponding to all the new query segments, and taking the obtained historical data as a query result.
Based on the data query method and device provided by the embodiment of the invention, the method comprises the following steps: receiving a first query request and extracting a query bit number in the first query request; inquiring a storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring historical data integrity information on the inquiry node within the starting time and the ending time; dividing the first query request into a plurality of second query requests according to the storage group corresponding to the query bit number, wherein the query bit number contained in each second query request is in the same storage group; for each second query request, dividing the second query request into a plurality of third query requests according to the query starting and ending time to obtain query segments corresponding to the third query requests, wherein each query segment corresponds to a time period; and performing data query on each query node based on the historical data integrity information and the query segment on each query node to obtain a query result. In the scheme, the first query request is divided into multiple stages according to the storage grouping and the query starting and stopping time by utilizing the query bit number to obtain the query segment corresponding to the third query request, and data query is performed on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is also improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data query method according to an embodiment of the present invention;
fig. 2 is an application scenario diagram for dividing a query request according to a storage packet corresponding to a query bit number according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of another data query method according to an embodiment of the present invention;
fig. 4 is an application scenario diagram for dividing query requests according to query start-stop times according to an embodiment of the present invention;
fig. 5 is a schematic flowchart of another data query method according to an embodiment of the present invention;
FIG. 6 is an application scenario diagram illustrating incomplete historical data within a query segment at a partial query node according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating an application scenario in which historical data in a query fragment at a partial query node is incomplete according to an embodiment of the present invention;
fig. 8 is an application scenario diagram illustrating that historical data in query segments on all query nodes are incomplete according to the embodiment of the present invention;
fig. 9 is a schematic structural diagram of a data query apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
As known in the background art, in the prior art, a data query method is low in query efficiency, or queried data is incomplete, and queried data is not reliable, so that efficiency of data query, integrity of data, and reliability of data cannot be guaranteed.
Therefore, in the present invention, the first query request is divided according to the storage grouping and the query start-stop time by using the query bit number to obtain the query segment corresponding to the third query request, and the query result is obtained by performing data query on each query node according to the storage information and the query segment corresponding to the query bit number, so that the query efficiency is improved, the integrity of the query data is ensured, and the reliability of the query data is also improved.
As shown in fig. 1, a schematic flow chart of a data query method provided in an embodiment of the present invention is shown, where the data query method is applicable to a distributed time series database, and mainly includes the following steps:
step S101: and receiving the first query request and extracting the query bit number in the first query request.
It should be noted that a query request generally includes a query number (i.e., a query object), a query start time, a query end time, a query type, and a desired number of results.
The query types include original query data, sampling, statistics and the like.
Preferably, extended filtering information is further included, for example, the filtering information is used to indicate data filtered to \8230;, from the original data, but the filtering information is not limited thereto.
The first query request is an external query request received by the distributed system.
In the process of implementing step S101 specifically, the distributed system receives the first query request, and at this time, the first query request only has the query bit number name and has no other message, so all query bit numbers are extracted from the first query request.
Step S102: and inquiring the storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring the historical data integrity information on the inquiry node within the starting time and the ending time.
In step S102, the step of querying the storage group corresponding to the bit number means that all the query bit numbers in the distributed system are divided into a plurality of groups according to a predetermined rule, and the data is stored in a group unit.
The query start-stop time is a query start time and a query end time.
In the process of implementing step S102 specifically, a storage group corresponding to a query bit number that is the same as the query bit number is queried in the distributed system based on the query bit number, and the historical data integrity information on the query node within the starting time and the ending time is queried.
It should be noted that the distributed system is a unified whole built on a network by a group of independent computers. The distributed system can store the inquiry bit and relevant information in advance, wherein the relevant information comprises a storage group corresponding to the inquiry bit number and historical data integrity information on the inquiry node within the inquiry starting and ending time.
Step S103: and dividing the first query request into a plurality of second query requests according to the storage groups corresponding to the query bit numbers.
In step S103, the query bit numbers included in each second query request are in the same memory packet.
The second query request is resolved internally within the distributed system and is an internal query request.
In the process of implementing step S103 specifically, on the basis of querying the storage packet corresponding to the query bit number, the first query request is divided into a plurality of second query requests according to the storage packet corresponding to the query bit number.
For example, if the first query request includes 10 query bit numbers, 5 query bit numbers are in the same storage group, 3 query bit numbers are in the same storage group, and 2 query bit numbers are in the same storage group, the first query request may be divided into 3 second query requests according to the storage group corresponding to the query bit number.
For another example, as shown in fig. 2, the first query request includes 9 query bit numbers (i.e., query objects), and the first query request may be divided into 3 second query requests according to the storage packet in which the query bit numbers are located, specifically, into Gp1, gp2, and Gp3 shown in fig. 2. That is, gp1, gp2, and Gp3 correspond to one second query request, respectively.
Step S104: and for each second query request, dividing the second query request into a plurality of third query requests according to the query start-stop time to obtain a plurality of query segments corresponding to the third query requests.
In step S104, each query segment corresponds to a time period.
The third query request is solved internally in the distributed system and is an internal query request.
In the process of implementing step S104 specifically, for each second query request, the query start-stop time is used to divide the second query request differently, when the query start-stop time is greater than a preset threshold, the query start-stop time is used to set a time period preset value, the second query request is divided into a plurality of third query requests according to the time period preset value, so as to obtain query segments corresponding to each query node, where each query segment includes all query bit numbers.
When the query start-stop time is less than the preset threshold value, the query start time and the query end time cannot be determined at the moment, the second query request is split into a plurality of third query requests according to the number of the query bit numbers, each query bit number becomes a third query request at the moment, a query segment corresponding to each query node is obtained, and the query segments at least comprise one query bit number.
Step S105: and performing data query on each query node based on the historical data integrity information and the query segments on each query node to obtain a query result.
In step S105, the query result is complete data obtained by combining the queried data.
In the process of implementing step S105 specifically, in the distributed system, historical data integrity information exists on each query node, data query is performed on each query node based on the query segment, the historical data integrity information of the query segment on each query node is queried, the historical data in the query segment on each query node is obtained and merged to obtain a query result, and the query result is fed back according to the query requirement.
According to the data query method provided by the embodiment of the invention, a first query request is received, and a query bit number in the first query request is extracted; inquiring a storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring historical data integrity information on the inquiry node within the starting time and the ending time; dividing the first query request into a plurality of second query requests according to a storage group corresponding to the query bit number, wherein the query bit number contained in each second query request is in the same storage group; for each second query request, dividing the second query request into a plurality of third query requests according to the query starting and stopping time to obtain query segments corresponding to the third query requests, wherein each query segment corresponds to a time period; and performing data query on each query node based on the historical data integrity information and the query segments on each query node to obtain a query result. According to the scheme, the first query request is divided into multiple stages according to the storage grouping and the query starting and stopping time by utilizing the query bit number to obtain the query segment corresponding to the third query request, and data query is performed on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is improved.
Based on the data query method shown in fig. 1 in the above embodiment of the present invention, as shown in fig. 3, a schematic flow chart of another data query method provided in the embodiment of the present invention is shown, where the data query method mainly includes the following steps:
step S301: and receiving a first query request and extracting a query bit number in the first query request.
Step S302: and inquiring the storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring the historical data integrity information on the inquiry node within the starting time and the ending time.
Step S303: and dividing the first query request into a plurality of second query requests according to the storage groups corresponding to the query bit numbers.
The execution principle and process of steps S301 to S303 are the same as the execution principle and process of steps S101 to S103 disclosed in fig. 1, and reference may be made to these steps, which are not described again here.
Step S304: for each second query request, determining whether the query start-stop time is greater than a predetermined threshold, if so, performing step S305, and if not, performing step S306.
Step S305: and dividing the second query request into a plurality of third query requests according to a preset time period value to obtain a query segment corresponding to each query node.
In step S305, each query segment contains all query bit numbers.
All time period preset values are combined into a complete query start-stop time.
In the process of implementing step S305 specifically, it is determined that the query start-stop time is greater than the preset threshold, a time period preset value is set by using the query start time and the query end time, and the second query request is segmented according to the set time period preset value, so as to obtain a query segment corresponding to each query node.
For example, as shown in fig. 4, the query start-stop time is t1-t4, the time period preset values are t1-t2, t2-t3, and t3-t4, for the second query request corresponding to Gp1, the second query request is divided into 3 third query requests according to the time period preset values, and the 3 third query requests are distributed to the query nodes, so as to obtain the query segments corresponding to each query Node, that is, the query segment corresponding to the t1-t2 time period of Node1, the query segment corresponding to the t2-t3 time period of Node2, and the query segment corresponding to the t3-t4 time period of Node 3.
Step S306: and splitting the second query request into a plurality of third query requests according to the query bit number to obtain a query segment corresponding to each query node.
In step S306, the query segment includes at least one query bit number.
In the process of implementing step S306 specifically, it is determined that the query start-stop time is less than the preset threshold, and the second query request is split into a plurality of third query requests according to the number of the query bit numbers, that is, one query bit number may be used as one third query request, so as to obtain a query segment corresponding to each query node.
For example, the second query request has 3 query bit numbers, the second query request is divided into 3 third query requests according to the query bit numbers, and the 3 third query requests are distributed to the query nodes, where each query node corresponds to one query segment.
For another example, as shown in fig. 4, the query start time and the query end time cannot be determined within the time period t1-t4, the number of the query bit numbers is 3, and for the second query request corresponding to Gp1, the second query request is split into 3 third query requests according to the number of the query bit numbers, and the 3 third query requests are distributed to the query nodes, so as to obtain the query segments corresponding to each query Node, that is, the query segments corresponding to one time period t1-t4 for Node1, node2, and Node3, respectively.
Step S307: and performing data query on each query node based on the historical data integrity information and the query segments on each query node to obtain a query result.
The execution principle and process of step S307 are the same as the execution principle and process of step S105 disclosed in fig. 1, and reference may be made to the execution principle and process, which are not described herein again.
According to the data query method provided by the embodiment of the invention, the first query request is divided into the second query requests according to the storage groups by utilizing the query bit number, when the query starting and stopping time is determined to be greater than the preset threshold value, the second query request is correspondingly segmented to obtain the query segment corresponding to the third query request, and data query is performed on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is also improved.
Based on the data query method shown in fig. 1 and fig. 3 in the foregoing embodiment of the present invention, as shown in fig. 5, a schematic flow chart of another data query method provided in the embodiment of the present invention is shown, where the data query method mainly includes the following steps:
step S501: and receiving the first query request and extracting the query bit number in the first query request.
Step S502: and inquiring the storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring the historical data integrity information on the inquiry node within the starting time and the ending time.
Step S503: and dividing the first query request into a plurality of second query requests according to the storage groups corresponding to the query bit numbers.
The execution principle and process of the above steps S501 to S503 are the same as the execution principle and process of the steps S101 to S103 disclosed in fig. 1, and reference may be made to these steps, which are not described herein again.
Step S504: for each second query request, determining whether the query start-stop time is greater than a preset threshold, if so, performing step S505, and if not, performing step S512.
Step S505: and dividing the second query request into a plurality of third query requests according to a preset time period value to obtain a query segment corresponding to each query node.
The principle and process of step S505 of step S504 are the same as the execution principle and process of steps S304 to S305 disclosed in fig. 3, and it can be referred to, and are not described herein again.
Step S506: and for each query node, judging whether the historical data in the query segment on the query node is complete or not based on the historical data integrity information on the query node.
In the process of implementing step S506 specifically, for each query node, it is determined whether the historical data in the query segment on the query node is complete based on the integrity information of the historical data on the query node, if the historical data in the query segments on all the query nodes are complete, step S507 is executed, if the historical data in the query segments on some query nodes are incomplete, step S508 is executed, and if the historical data in the query segments on all the query nodes are incomplete, step S510 is executed.
Step S507: and combining the historical data in the query segments on each query node to obtain a query result.
It should be noted that, when merging the historical data, the queried historical data needs to be merged according to the query type.
In the process of implementing step S507 specifically, it is determined that the historical data in the query segments on all the query nodes are complete, the historical data in the query segments on each query node is obtained, and the historical data in the query segments on each query node is merged to obtain a query result.
For example, there are 3 query nodes, and the historical data in the query segments on the 3 query nodes are all complete, and at this time, the historical data in the query segments on the 3 query nodes need to be merged to obtain complete historical data within the query start-stop time, and the complete historical data within the query start-stop time is the query result.
Step S508: and determining a new query segment on the partial query node based on the query start-stop time and the time period corresponding to other query segments with complete historical data.
In step S508, the time period corresponding to the new query segment is a difference between the query start-stop time and the time periods corresponding to the other query segments with complete historical data, that is, the time period corresponding to the other query segments with complete historical data is subtracted from the query start-stop time to obtain the time period corresponding to the new query segment.
In the process of implementing step S508 specifically, it is determined that the historical data in the query segment on the partial query node is incomplete, and other query segments with complete historical data on the partial query node are determined, and a new query segment on the partial query node is obtained according to the query start-stop time and the determined other query segments with complete historical data.
Step S509: and combining the historical data in other query segments and the historical data in the new query segments on part of the query nodes to obtain a query result.
In the process of implementing step S509 specifically, the historical data in other query segments and the historical data in new query segments on part of the query nodes are obtained, and the obtained historical data in other query segments and the historical data in new query segments on part of the query nodes are merged to obtain a query result.
For example, step S508 and step S509 are illustrated, as shown in fig. 6 and fig. 7, fig. 6 is an application scenario diagram of incomplete historical data in a query segment on a partial query node according to an embodiment of the present invention, and fig. 7 is an application scenario diagram of incomplete historical data in a query segment on another partial query node according to an embodiment of the present invention.
In fig. 6, there are 3 query nodes, when a query segment performs data query on 3 query nodes, historical data exists in the query segment corresponding to the 3 query nodes, but the historical data in the query segment corresponding to the query Node3 is incomplete, as shown in fig. 6, the query Node3 displays the query segment of Bad, which indicates that the historical data in the query segment corresponding to the query Node3 is incomplete, at this time, it is necessary to determine other query segments with complete historical data corresponding to the query Node3, and determine other query segments with complete historical data corresponding to the query Node3 as t2-t3.
According to the query start-stop time and the time period corresponding to other query segments with complete historical data corresponding to the query Node3, the new query segments on the query Node1 and the query Node2 are determined, as shown in fig. 7, it is determined that the new query segment on the query Node1 is T0' -T2, the new query segment on the query Node2 is T2-T3, and the query segment T0' -T3 corresponding to the query Node3 is a query segment with incomplete history data, i.e. the query segment with Bad shown in fig. 7, therefore, the historical data in other query segments with complete historical data on the query Node3 (that is, the historical data in the query segment T0-T0' with complete historical data on the query Node 3) and the historical data in new query segments corresponding to the query Node1 and the query Node2 are obtained, and the historical data in other query segments with complete historical data corresponding to the query Node3 and the historical data in new query segments corresponding to the query Node1 and the query Node2 are merged to obtain complete historical data in the query starting and ending time, so that the query result is obtained.
Step S510: and determining a new query segment on each query node based on the query starting and ending time, the time period corresponding to each query segment and a preset partition strategy.
In the process of implementing step S510 specifically, it is determined that historical data in query segments on all query nodes are incomplete, a time period corresponding to each query segment is determined, and a new query segment on each query node is determined according to the query start/stop time, the time period corresponding to each query segment, and a preset partition policy.
Step S511: and combining the historical data in the new query segment on each query node to obtain a query result.
In the process of implementing step S511 specifically, the historical data in the new query segment on each query node is acquired, and the acquired historical data in the new query segment on each query node is merged to obtain a query result.
For example, step S510 and step S511 are illustrated, as shown in fig. 8, which is an application scenario diagram provided by the embodiment of the present invention, where historical data in query segments on all query nodes are incomplete.
In fig. 8, there are 3 query nodes, and when a query segment performs data query on the 3 query nodes, historical data in the query segments corresponding to the 3 query nodes are incomplete, that is, the query segment of Bad is displayed on the query Node1 in fig. 8, the query segment of Bad is displayed on the query Node2, and the query segment of Bad is displayed on the query Node3, and according to the query start-stop time and the time period corresponding to each query segment corresponding to the 3 query nodes, the setting division strategy at this time is: the time period of the next section of normal data of the query Node1, the time period of the previous section of normal data of the query Node 2/the query Node3, and the query Node 1/query Node 2/query Node3 are abnormal time periods, that is, the corresponding new query segments on the 3 query nodes are determined.
According to the set partitioning strategy, it can be obtained that the last selected segment is abnormal, so that the historical data in the corresponding new query segments on the 3 query nodes needs to be merged to obtain the complete historical data in the query starting and ending time, namely the query result.
Step S512: and splitting the second query request into a plurality of third query requests according to the query bit number to obtain a query segment corresponding to each query node.
The execution principle and process of step S512 are the same as those of step S306 disclosed in fig. 3, and reference may be made to the execution principle and process, which are not described herein again.
Step S513: and for each query node, judging whether the historical data in the query segment on the query node is complete or not based on the historical data integrity information on the query node.
In the process of implementing step S513 specifically, for each query node, it is determined whether the historical data in the query segment on the query node is complete based on the historical data integrity information on the query node, if the historical data in the query segment on any query node is complete, step S314 is executed, and if the historical data in the query segments on all query nodes is incomplete, step S315 is executed.
Step S514: and acquiring historical data in the query segment as a query result.
In the process of implementing step S514 specifically, it is determined that the historical data in the query segment on any query node is complete, the historical data in the query segment on any query node is obtained, and the obtained historical data is used as a query result.
Step S515: and determining the segments with complete historical data on all the query nodes as new query segments.
In the process of implementing step S515 specifically, it is determined that the historical data in the query segments on all the query nodes are incomplete, the historical data in the query segments on all the query nodes are obtained, and the segments with complete historical data on all the query nodes are determined as new query segments.
Step S516: and performing deduplication processing on the historical data corresponding to all the new query segments, and taking the obtained historical data as a query result.
In the process of implementing step S516 specifically, repeated historical data among all new query segments is found out, and all repeated historical data is subjected to deduplication processing to obtain final historical data, and the obtained final historical data is used as a query result.
According to the data query method provided by the embodiment of the invention, the first query request is divided into the second query requests according to the storage groups by utilizing the query bit number, the second query requests are correspondingly segmented according to the query starting and stopping time to obtain a plurality of query segments, and for different segmentation processes of the second query requests, data query is carried out on each query node according to the historical data integrity information on each query node and the obtained query segments to obtain a query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is also improved.
Corresponding to the data query method shown in the above embodiment of the present invention, an embodiment of the present invention further provides a data query device, as shown in fig. 9, the data query device is suitable for a distributed time sequence database, and the data query device includes: a processing module 91, a first query module 92, a first partitioning module 93, a second partitioning module 94, and a second query module 95.
The extracting module 91 is configured to receive the first query request and extract the query bit number in the first query request.
And the first query module 92 is used for querying the storage group corresponding to the query bit number based on the query bit number and querying the historical data integrity information on the query node within the starting time and the ending time.
The first dividing module 93 is configured to divide the first query request into a plurality of second query requests according to the storage group corresponding to the query bit number, where the query bit number included in each second query request is in the same storage group.
The second dividing module 94 is configured to, for each second query request, divide the second query request into a plurality of third query requests according to the query start-stop time, so as to obtain query segments corresponding to the third query requests, where each query segment corresponds to a time period.
And a second query module 95, configured to perform data query on each query node based on the historical data integrity information and the query segment on each query node, so as to obtain a query result.
It should be noted that, the specific principle and the execution process of each module in the data query device disclosed in the above embodiment of the present invention are the same as the data query method implemented in the above embodiment of the present invention, and reference may be made to the corresponding parts in the data query method disclosed in the above embodiment of the present invention, which are not described again here.
According to the data query device provided by the embodiment of the invention, the query bit number in the first query request is extracted by receiving the first query request; inquiring a storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring historical data integrity information on the inquiry node within the starting time and the ending time; dividing the first query request into a plurality of second query requests according to a storage group corresponding to the query bit number, wherein the query bit number contained in each second query request is in the same storage group; for each second query request, dividing the second query request into a plurality of third query requests according to the query starting and stopping time to obtain query segments corresponding to the third query requests, wherein each query segment corresponds to a time period; and performing data query on each query node based on the historical data integrity information and the query segments on each query node to obtain a query result. According to the scheme, the first query request is divided into multiple stages according to the storage grouping and the query starting and stopping time by utilizing the query bit number to obtain the query segment corresponding to the third query request, and data query is performed on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is improved.
Optionally, based on the second dividing module 94 shown in fig. 9, the second dividing module 94 is specifically configured to, for each second query request, divide the second query request into a plurality of third query requests according to a preset time period value if the query start-stop time is greater than a preset threshold value, to obtain a query segment corresponding to each query node, where each query segment includes all query bit numbers.
According to the data query device provided by the embodiment of the invention, the first query request is divided into the second query requests according to the storage groups by utilizing the query bit number, when the query starting and stopping time is determined to be greater than the preset threshold value, the second query request is correspondingly segmented to obtain the query segment corresponding to the third query request, and data query is carried out on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is also improved.
Optionally, based on the second query module 95 shown in fig. 9, the second query module 95 further includes: the device comprises a first judgment unit, a first merging unit, a second merging unit and a third merging unit.
The first judging unit is used for judging whether the historical data in the query segments on the query nodes are complete or not according to the historical data integrity information on the query nodes, if the historical data in the query segments on all the query nodes are complete, the first merging unit is executed, if the historical data of part of the query nodes in the query segments are incomplete, the second merging unit is executed, and if the historical data of all the query nodes in the query segments are incomplete, the third merging unit is executed.
And the first merging unit is used for merging the historical data in the query segments on each query node to obtain a query result.
The second merging unit is used for determining new query segments on partial query nodes based on the query start-stop time and the time periods corresponding to other query segments with complete historical data; and combining the historical data in other query segments and the historical data in the new query segments on part of the query nodes to obtain a query result.
The third merging unit is used for determining a new query segment on each query node based on the query starting and ending time, the time period corresponding to each query segment and a preset dividing strategy; and merging the historical data in the new query segment on each query node to obtain a query result.
According to the data query device provided by the embodiment of the invention, the query bit number is utilized to divide the first query request into multiple stages according to the storage grouping and the query starting and stopping time to obtain the query segment corresponding to the third query request, and data query is performed on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is also improved.
Optionally, based on the second dividing module 94 shown in fig. 9, the second dividing module 94 is further specifically configured to, for each second query request, split the second query request into a plurality of third query requests according to the query bit number if the query start-stop time is smaller than a preset threshold, and obtain a query segment corresponding to each query node, where the query segment includes at least one query bit number.
According to the data query device provided by the embodiment of the invention, the first query request is divided into the second query requests according to the storage groups by utilizing the query bit number, when the query starting and stopping time is determined to be less than the preset threshold value, the corresponding segmentation processing is carried out on the second query request to obtain the query segment corresponding to the third query request, and the data query is carried out on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the credibility of the query data is also improved.
Optionally, based on the second query module 95 shown in fig. 9, the second query module 95 further includes: a second judging unit, a fourth merging unit and a fifth merging unit.
And the second judging unit is used for judging whether the historical data in the query segments on the query nodes are complete or not according to the historical data integrity information on the query nodes, executing the fourth merging unit if the historical data in the query segments on any query node are complete, and executing the fifth merging unit if the historical data of all the query nodes in the query segments are incomplete.
And the fourth merging unit is used for acquiring the historical data in the query segment as a query result.
The fifth merging unit is used for determining the segments with complete historical data on all the query nodes as new query segments; and performing deduplication processing on the historical data corresponding to all the new query segments, and taking the obtained historical data as a query result.
According to the data query device provided by the embodiment of the invention, the query bit number is utilized to divide the first query request into multiple stages according to the storage grouping and the query starting and stopping time to obtain the query segment corresponding to the third query request, and data query is performed on each query node according to the historical data integrity information on each query node and the obtained query segment to obtain the query result, so that the query efficiency is improved, the integrity of the query data is ensured, and meanwhile, the reliability of the query data is also improved.
All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. A data query method, adapted to a distributed time series database, the method comprising:
receiving a first query request, and extracting a query bit number in the first query request;
inquiring a storage group corresponding to the inquiry bit number based on the inquiry bit number, and inquiring historical data integrity information on the inquiry node within the starting time and the ending time;
dividing the first query request into a plurality of second query requests according to the storage group corresponding to the query bit number, wherein the query bit number contained in each second query request is in the same storage group;
for each second query request, dividing the second query request into a plurality of third query requests according to the query starting and stopping time to obtain query segments corresponding to the third query requests, wherein each query segment corresponds to a time period;
based on the historical data integrity information and the query segments on each query node, performing data query on each query node to obtain a query result, including:
for each query node, judging whether the historical data in the query segment on the query node is complete or not based on the historical data integrity information on the query node; if the historical data in the query segments on all the query nodes are complete, combining the historical data in the query segments on each query node to obtain a query result; if the historical data in the query segments on the partial query nodes are incomplete, determining new query segments on the partial query nodes based on the query starting and ending time and time periods corresponding to other query segments with complete historical data; combining the historical data in the other query segments with the historical data in the new query segments on the partial query nodes to obtain a query result; if the historical data in the query segments on all the query nodes are incomplete, determining a new query segment on each query node based on the query starting and ending time, the time period corresponding to each query segment and a preset dividing strategy; merging the historical data in the new query segment on each query node to obtain a query result;
alternatively, the first and second liquid crystal display panels may be,
for each query node, judging whether the historical data in the query segment on the query node is complete or not based on the historical data integrity information on the query node; if the historical data in the query segment on any query node is complete, acquiring the historical data in the query segment as a query result; if the historical data in the query segments on all the query nodes are incomplete, determining the segments with complete historical data on all the query nodes as new query segments; and performing deduplication processing on the historical data corresponding to all the new query segments, and taking the obtained historical data as a query result.
2. The method of claim 1, wherein for each of the second query requests, the dividing the second query request into a plurality of third query requests according to the query start-stop time to obtain a query segment corresponding to the third query request comprises:
and for each second query request, if the query start-stop time is greater than a preset threshold value, dividing the second query request into a plurality of third query requests according to a preset time period value to obtain a query segment corresponding to each query node, wherein each query segment comprises all query bit numbers.
3. The method of claim 1, wherein for each of the second query requests, the dividing the second query request into a plurality of third query requests according to the query start-stop time to obtain a query segment corresponding to the third query request comprises:
and for each second query request, if the query start-stop time is less than a preset threshold value, splitting the second query request into a plurality of third query requests according to the query bit number to obtain a query segment corresponding to each query node, where the query segment at least includes one query bit number.
4. A data query apparatus, adapted for use with a distributed time series database, the apparatus comprising:
the extraction module is used for receiving a first query request and extracting a query bit number in the first query request;
the first query module is used for querying the storage group corresponding to the query bit number based on the query bit number and querying the historical data integrity information on the query node within the starting time and the ending time;
the first dividing module is used for dividing the first query request into a plurality of second query requests according to the storage group corresponding to the query bit number, and the query bit number contained in each second query request is in the same storage group;
the second dividing module is used for dividing each second query request into a plurality of third query requests according to the query starting and stopping time to obtain query segments corresponding to the third query requests, and each query segment corresponds to a time period;
the second query module is used for performing data query on each query node based on the historical data integrity information and the query segments on each query node to obtain a query result;
the second query module comprises: the first determining unit, the first merging unit, the second merging unit, and the third merging unit, or the second querying module includes: a second judgment unit, a fourth merging unit and a fifth merging unit;
the first judging unit is configured to judge, for each query node, whether historical data in the query segment on the query node is complete based on the historical data integrity information on the query node, execute a first merging unit if the historical data in the query segment on all the query nodes is complete, execute a second merging unit if the historical data in the query segment of part of the query nodes is incomplete, and execute a third merging unit if the historical data in the query segment of all the query nodes is incomplete;
the first merging unit is configured to merge historical data in the query segment on each query node to obtain a query result;
the second merging unit is configured to determine a new query segment on the part of the query nodes based on the query start-stop time and a time period corresponding to another query segment with complete historical data; combining the historical data in the other query segments with the historical data in the new query segments on the partial query nodes to obtain a query result;
the third merging unit is configured to determine a new query segment on each query node based on the query start-stop time, a time period corresponding to each query segment, and a preset partition policy;
combining historical data in the new query segments on each query node to obtain a query result;
the second judging unit is configured to judge, for each query node, whether the historical data in the query segment on the query node is complete based on the historical data integrity information on the query node, execute a fourth merging unit if the historical data in the query segment on any query node is complete, and execute a fifth merging unit if the historical data in the query segment of all the query nodes is incomplete;
the fourth merging unit is configured to obtain historical data in the query segment as a query result;
the fifth merging unit is used for determining the segments with complete historical data on all the query nodes as new query segments; and performing deduplication processing on the historical data corresponding to all the new query segments, and taking the obtained historical data as a query result.
5. The apparatus according to claim 4, wherein the second partitioning module is specifically configured to:
and for each second query request, if the query start-stop time is greater than a preset threshold, dividing the second query request into a plurality of third query requests according to a preset time period value to obtain a query segment corresponding to each query node, wherein each query segment comprises all query bit numbers.
6. The apparatus of claim 4, wherein the second partitioning module is further specifically configured to:
and for each second query request, if the query start-stop time is less than a preset threshold value, splitting the second query request into a plurality of third query requests according to the query bit number to obtain a query segment corresponding to each query node, where the query segment at least includes one query bit number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110717857.5A CN113342856B (en) | 2021-06-28 | 2021-06-28 | Data query method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110717857.5A CN113342856B (en) | 2021-06-28 | 2021-06-28 | Data query method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113342856A CN113342856A (en) | 2021-09-03 |
| CN113342856B true CN113342856B (en) | 2022-10-18 |
Family
ID=77479199
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110717857.5A Active CN113342856B (en) | 2021-06-28 | 2021-06-28 | Data query method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113342856B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8458285B2 (en) * | 2008-03-20 | 2013-06-04 | Post Dahl Co. Limited Liability Company | Redundant data forwarding storage |
| WO2015048925A1 (en) * | 2013-10-03 | 2015-04-09 | Huawei Technologies Co., Ltd. | A method of optimizing queries execution on a data store |
| CN108737473B (en) * | 2017-04-20 | 2019-12-20 | 贵州白山云科技股份有限公司 | Data processing method, device and system |
| CN109117428B (en) * | 2017-06-26 | 2020-12-08 | 北京嘀嘀无限科技发展有限公司 | Data storage method and device, and data query method and device |
| CN110619008A (en) * | 2019-09-17 | 2019-12-27 | 北京百度网讯科技有限公司 | Database query method and device, electronic equipment and storage medium |
-
2021
- 2021-06-28 CN CN202110717857.5A patent/CN113342856B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113342856A (en) | 2021-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106406896B (en) | Block chain block building method based on parallel Pipeline technology | |
| US10331642B2 (en) | Data storage method and apparatus | |
| EP3519989A1 (en) | Query optimized distributed ledger system | |
| US20160226745A1 (en) | Estimating latency of an application | |
| CN110022221B (en) | System interface data monitoring method, device and system | |
| CN110661658A (en) | Node management method and device of block chain network and computer storage medium | |
| CN110597918A (en) | Account management method and device and computer readable storage medium | |
| CN107656705B (en) | Computer storage medium and data migration method, device and system | |
| CN106878038B (en) | Fault positioning method and device in communication network | |
| CN109981326B (en) | Method and device for positioning household broadband sensing fault | |
| CN110795499A (en) | Cluster data synchronization method, device and equipment based on big data and storage medium | |
| CN110647531A (en) | Data synchronization method, device, equipment and computer readable storage medium | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN105447168A (en) | Method for restoring and recombining fragmented files in MP4 format | |
| CN110505084B (en) | Block chain packing node consensus electing method | |
| CN109299116B (en) | Data synchronization method, device, equipment and readable storage medium | |
| CN113342856B (en) | Data query method and device | |
| CN104778252A (en) | Index storage method and index storage device | |
| CN114791927A (en) | Data analysis method and device | |
| CN111552701B (en) | Method for determining data consistency in distributed cluster and distributed data system | |
| CN110442439B (en) | Task process processing method and device and computer equipment | |
| CN112436962A (en) | Block chain consensus network dynamic expansion method, electronic device, system and medium | |
| CN110990360A (en) | File synchronization method based on network storage device and related components | |
| CN114138786A (en) | Method, device, medium, product and equipment for duplicate removal of online transaction message | |
| CN110569231B (en) | Data migration method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |