CN116680295A - Method, system and device for processing data by multiple databases - Google Patents
Method, system and device for processing data by multiple databases Download PDFInfo
- Publication number
- CN116680295A CN116680295A CN202310653188.9A CN202310653188A CN116680295A CN 116680295 A CN116680295 A CN 116680295A CN 202310653188 A CN202310653188 A CN 202310653188A CN 116680295 A CN116680295 A CN 116680295A
- Authority
- CN
- China
- Prior art keywords
- data
- queried
- target
- query
- target object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012545 processing Methods 0.000 title claims abstract description 51
- 230000006870 function Effects 0.000 claims description 24
- 238000004364 calculation method Methods 0.000 claims description 22
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 238000012216 screening Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 abstract description 15
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000004590 computer program Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000008030 elimination Effects 0.000 description 4
- 238000003379 elimination reaction Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000002071 nanotube Substances 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a system and a device for processing data by a multi-database. Wherein the method comprises the following steps: receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one. The application solves the technical problems of low data query efficiency and low query response speed caused by poor throughput and low expandability of the traditional relational database in the related technology.
Description
Technical Field
The application relates to the field of data processing, in particular to a method, a system and a device for processing data by a plurality of databases.
Background
In a telecom operator network, performance data generated by the operation of network equipment generally has the characteristics of large data volume, high concurrency and strong real-time performance, the data growth speed is extremely high, and when the data is stored and processed, the throughput and the expandability of a database are required to be high, and the traditional relational database is difficult to meet. However, as the range of the device nanotubes increases, the read-write pressure of the database increases, and the storage mode which depends on OpenTSDB only gradually reaches the performance bottleneck, which mainly has the following problems:
1. The existing network equipment has huge scale, a plurality of logical resources such as ports, subinterfaces, management domains and the like, the daily data volume exceeds billions, and meanwhile, high concurrent data writing and query are considered, the processing speed of a CPU (Central processing Unit) of a server reaches a performance bottleneck, the QPS (quality control System) can only reach tens of, and the query response speed is slow;
2. in OpenTSDB, device IP and port names are stored as tagv in metadata, but because the association relationship between the device and the port cannot be stored in tagv, when a specified port of a specified device is queried, a program can match appropriate data one by one from a cartesian product of all device IP and port names, the query range grows exponentially, and the data query efficiency is greatly affected;
3. in the use scene of the equipment performance data, the user is generally more concerned about the recently generated real-time data, the real-time data is used for monitoring the current running condition of the equipment, and the requirement on the query response speed of the hot data is higher. For the historical data with long storage time, the query frequency of the user is generally low, the historical data is mainly used for visualization or predictive analysis, and the response speed requirement on the query is low. However, the current OpenTSDB does not have a hierarchical storage mechanism for cold and hot data, and the cold and hot data share the same set of query storage mechanism, so that the query performance of the hot data is reduced, and unnecessary storage cost and resource waste are caused for query storage of the cold data.
In view of the above problems, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the application provides a method, a system and a device for processing data by multiple databases, which at least solve the technical problems of low data query efficiency and low query response speed caused by low throughput and low expandability of the traditional relational databases in the related technology.
According to an aspect of an embodiment of the present application, there is provided a method of processing data by a multi-database, including: receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
Optionally, determining, according to the identification information, the target data type to which the data to be queried belongs, including: determining the query frequency or time range of the data to be queried corresponding to the identification information in a historical period, wherein the time range is used for indicating the time interval between the last time of receiving a historical data query request of the data to be queried and the current time of receiving the data query request; under the condition that the query frequency is greater than the preset frequency or the time range is smaller than the preset duration, determining the data to be queried as hot data; and under the condition that the query frequency is smaller than the preset frequency or the time range is larger than the preset duration, determining the data to be queried as cold data.
Optionally, determining a data query interface corresponding to the target data type, and returning the data queried by the data query interface to the target object includes: calling a data query interface of the OpenTSDB database under the condition that the target data type is cold data; acquiring all sub-table information of the index in the time range from an OpenTSDB database according to the index transmitted by the target object and the selected time range, wherein a scan function in the OpenTSDB database is used for assigning the equipment IP corresponding to the target object to the start time parameter and the end time end parameter; judging whether a function calculation requirement exists in parameters input from the target object, carrying out logic operation according to the calculation requirement, and returning a calculation result to the target object.
Optionally, before acquiring all the sub-table information of the index in the time range, the method further includes: and carrying out query assignment on the device IP by using regular matching of a filter, and changing the query assignment into assignment in a start parameter and an end parameter, wherein the filter is only used for matching port names.
Optionally, determining a data query interface corresponding to the target data type, and returning the data queried by the data query interface to the target object includes: calling a data query interface of the Redis database under the condition that the target data is hot data; judging whether a function calculation requirement exists in parameters input from the target object, carrying out logic operation according to the calculation requirement, and returning a calculation result to a terminal corresponding to the target object.
Optionally, before invoking the data query interface of the Redis database, the method further comprises: determining a storage period of a Redis database; and acquiring the historical data written into the Redis database in the historical period, and deleting other data except for the storage period in the Redis database.
Optionally, after deleting other data than the storage period in the Redis database, the method further includes: determining data of each object requesting to be queried in a historical period, analyzing the data requesting to be queried in the historical period based on a page replacement algorithm, screening out new hot data, and writing the new hot data into a Redis database.
According to another aspect of an embodiment of the present application, there is also provided a system for processing data by a multi-database, including: the data writing module is provided with a Redis database corresponding to the stored hot data and an OpenTSDB database used for storing cold data; the data query module is used for receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried, and determines a target data type to which the data to be queried belongs according to the identification information, and the data type comprises one of the following: hot data, or cold data; and determining a data query interface corresponding to the target data type, and returning the structural data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
Optionally, a scan function in the OpenTSDB database assigns a device IP corresponding to the target object in the start and end parameters.
According to another aspect of the embodiment of the present application, there is also provided an apparatus for processing data by a multi-database, including: the receiving module is used for receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried; the query module is used for determining the target data type to which the data to be queried belongs according to the identification information, wherein the data type comprises one of the following: hot data, cold data; the determining module is used for determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
According to another aspect of the embodiment of the present application, there is also provided a nonvolatile storage medium, where the storage medium includes a stored program, and when the program runs, the storage medium is controlled to perform any one of methods for processing data by using multiple databases.
According to another aspect of the embodiment of the present application, there is also provided an electronic device, including: a processor; wherein the processor is configured to execute instructions to implement any one of a number of methods of processing data with the database.
In the embodiment of the application, a mode of separately storing cold and hot data is adopted, and a data query request of a target object is received, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; the method comprises the steps of determining a data query interface corresponding to a target data type, and returning result data queried by the data query interface to a target object in a unified data format, wherein the target data type corresponds to the data query interface one by one, so that the aim of classifying and storing cold and hot data based on databases of different types is fulfilled, the technical effects of improving the data processing throughput of the databases, improving the data query efficiency and the response speed are realized, and the technical problems of lower data query efficiency and lower query response speed caused by poor throughput, lower expandability of a traditional relational database in related technologies are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a flow diagram of a method of processing data with multiple databases according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a system for processing data with multiple databases according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a system for processing data with multiple databases according to an embodiment of the present application;
FIG. 4 is a flow chart of a method for processing data by multiple databases according to an embodiment of the application;
FIG. 5 is a schematic diagram of a real-time diversity storage flow of device operational data in an embodiment of the application;
FIG. 6 is a flow chart of real-time querying of device operational data in an embodiment of the present application;
FIG. 7 is a schematic diagram of an apparatus for processing data with multiple databases according to an embodiment of the present application;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For a better understanding of the related embodiments of the present application, technical terms or terms that may be related to the embodiments of the present application will now be explained:
qps: queries Per Second is meant to be "query rate per second", which is the number of queries a server can respond to per second, and is a measure of how much traffic a particular query server is handling in a given time.
Lru is an abbreviation for Least Recently Used, i.e., least recently used, is a common page replacement algorithm that selects the least recently used page to eliminate. The algorithm gives each page an access field for recording the time t that a page has elapsed since the last time it was accessed, and when a page has to be eliminated, the page with the largest t value, i.e. the least recently used page, of the existing pages is selected for elimination.
In accordance with an embodiment of the present application, there is provided a method embodiment for processing data in a database, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system, such as a set of computer executable instructions, and that although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
FIG. 1 is a method of processing data for multiple databases according to an embodiment of the present application, as shown in FIG. 1, the method comprising the steps of:
s102, receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried;
s104, determining a target data type to which the data to be queried belongs according to the identification information, wherein the data type comprises one of the following: hot data, cold data;
S106, determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
In the method for processing data by the multiple databases, a mode of storing cold and hot data is adopted, and a data query request of a target object is received, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; the method comprises the steps of determining a data query interface corresponding to a target data type, and returning result data queried by the data query interface to a target object in a unified data format, wherein the target data type corresponds to the data query interface one by one, so that the aim of classifying and storing cold and hot data based on databases of different types is fulfilled, the technical effects of improving the data processing throughput of the databases, improving the data query efficiency and the response speed are realized, and the technical problems of lower data query efficiency and lower query response speed caused by poor throughput, lower expandability of a traditional relational database in related technologies are solved.
In some embodiments of the present application, determining, according to the identification information, a target data type to which the data to be queried belongs may be implemented in the following manner: the query frequency or time range of the data to be queried corresponding to the identification information in the historical period can be determined, wherein the time range is used for indicating the time interval duration between the last time of receiving a historical data query request of the data to be queried and the time of receiving the data query request; under the condition that the query frequency is greater than the preset frequency or the time range is smaller than the preset duration, determining the data to be queried as hot data; and under the condition that the query frequency is smaller than the preset frequency or the time range is larger than the preset duration, determining the data to be queried as cold data.
In order to improve the processing efficiency of cold data, as an optional implementation manner, determining a data query interface corresponding to a target data type, and returning the data queried by the data query interface to the target object, where the determining includes: calling a data query interface of the OpenTSDB database under the condition that the target data type is cold data; acquiring all sub-table information of the index in the time range from an OpenTSDB database according to the index transmitted by the target object and the selected time range, wherein a scan function in the OpenTSDB database is used for assigning the equipment IP corresponding to the target object to the start time parameter and the end time end parameter; judging whether a function calculation requirement exists in parameters input from the target object, carrying out logic operation according to the calculation requirement, and returning a calculation result to the target object.
In some embodiments of the present application, in order to avoid the cartesian product problem, before all the sub-table information of the index in the time range is obtained, query assignment can be changed from query assignment of the device IP by using regular matching of the filter, where the filter is only used for matching port names, to assignment in the start parameter and the end parameter.
In order to improve the query efficiency of hot data, determining a data query interface corresponding to a target data type, and returning the data queried by the data query interface to a target object, wherein the method comprises the following steps: calling a data query interface of the Redis database under the condition that the target data is hot data; judging whether a function calculation requirement exists in parameters input from the target object, carrying out logic operation according to the calculation requirement, and returning a calculation result to a terminal corresponding to the target object.
In order to reduce occupation of the Redis database, the storage period of the Redis database can be determined before a data query interface of the Redis database is called; and acquiring the historical data written into the Redis database in the historical period, and deleting other data except for the storage period in the Redis database.
Optionally, after deleting other data than the storage period in the Redis database, the method further includes: determining data of each object requesting to be queried in a historical period, analyzing the data requesting to be queried in the historical period based on a page replacement algorithm, screening out new hot data, and writing the new hot data into a Redis database.
FIG. 2 is a system for processing data with multiple databases according to an embodiment of the present application, as shown in FIG. 2, the system comprising:
a data writing module 20, provided with a Redis database for storing hot data and an OpenTSDB database for storing cold data;
the data query module 22 is configured to receive a data query request of a target object, where the data query request at least carries identification information of data to be queried, and determine, according to the identification information, a target data type to which the data to be queried belongs, where the data type includes one of: hot data, or cold data; and determining a data query interface corresponding to the target data type, and returning the structural data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
In a system for processing data by multiple databases, a data writing module 20 is provided with a Redis database for storing hot data and an OpenTSDB database for storing cold data; the data query module 22 is configured to receive a data query request of a target object, where the data query request at least carries identification information of data to be queried, and determine, according to the identification information, a target data type to which the data to be queried belongs, where the data type includes one of: hot data, or cold data; the method comprises the steps of determining a data query interface corresponding to a target data type, and returning structural data queried by the data query interface to a target object in a unified data format, wherein the target data type corresponds to the data query interface one by one, so that the aim of classifying and storing cold and hot data based on databases of different types is fulfilled, the technical effects of improving the data processing throughput of the databases, improving the data query efficiency and the response speed are realized, and the technical problems of lower data query efficiency and lower query response speed caused by poor throughput, lower expandability of a traditional relational database in the related technology are solved.
Optionally, a scan function in the OpenTSDB database assigns a device IP corresponding to the target object in the start and end parameters.
FIG. 3 is a system for processing data in multiple databases according to one embodiment of the application, as shown in FIG. 3, the system comprising:
and a data writing module: after data is analyzed and put in storage, the OpenTSDB and the Redis are written in at the same time, and double-write storage is carried out;
OpenTSDB transformation module: according to the index and time of inquiring/writing, different sub-tables are obtained, if the sub-tables do not exist, a global lock is obtained, a corresponding sub-table is created, the Rowkey index of the HBASE bottom layer is optimized, a scan function is modified, regular matching inquiry of the equipment IP from the filter is changed into assignment at start and end, the filter only matches port names, and therefore the Cartesian product problem is optimized; the api/query interface of the OpenTSDB is correspondingly transformed according to the rules;
redis transformation module: modifying a Redis data structure based on the zset data set, and eliminating structural differences of OpenTSDB and Redis data query output; making a data elimination strategy, deleting historical data in Redis according to the data writing time, and deleting data with few inquiry times by using an LRU algorithm; defining a Redis data query interface according to the characteristic of the OpenTSDB returned data, wherein the Redis data query interface comprises a performance data query interface, a multi-index item query interface and a multi-time period query interface, and supports sum and avg function calculation;
And a data query module: and the packaging query interface judges whether the query belongs to cold data or hot data according to the time range and the query frequency of the query data of the user, and obtains the data from the OpenTSDB or the Redis respectively to provide real-time data query service.
FIG. 4 is a flowchart of a method for processing data by multiple databases according to an embodiment of the present application, as shown in FIG. 4, the process includes:
s1, performing double-write operation on the acquired and analyzed data, and simultaneously writing in an OpenTSDB and a Redis.
S2.1, entering the table according to the writing index and the writing time, and if the corresponding table does not exist, acquiring the global lock to create the sub-table.
S2.2, optimizing a Rowkey index of an HBASE bottom layer, modifying a scan function, changing regular matching query of equipment IP from a filter into assignment at start and end, wherein the filter only matches port names, thereby avoiding the occurrence of Cartesian product problem.
S2.2, correspondingly modifying an api/query interface of the OpenTSDB according to the rule in S2.1.
S3.1, modifying a Redis data structure based on the zset data set, and eliminating structural differences of OpenTSDB and Redis data query output.
And S3.2, making a data elimination strategy, deleting the historical data in the Redis according to the data writing time, and deleting the data with few inquiry times by using the LRU algorithm.
S3.3, defining a Redis data query interface according to the characteristic of the OpenTSDB returned data, wherein the Redis data query interface comprises a performance data query interface, a multi-index item query interface and a multi-time period query interface, and supports sum and avg function calculation.
S4, uniformly packaging the query interfaces of the OpenTSDB and the Redis.
S5, when a data query request of a user is received, judging whether the data is hot data or cold data according to the time of querying the data and the query frequency of the queried content.
S5.1, if the data is hot data, calling a query interface of Redis.
S5.2, if the data is cold data, calling a query interface of the OpenTSDB.
S5.3, the inquiry may involve multiple sub-tables, then the multiple sub-tables are inquired simultaneously, all returned data are acquired and then are processed by aggregation or other processing logic
And S6, returning the query result to the user in a unified data format.
The above technical scheme of the present application will be further described with reference to specific embodiments:
fig. 5 is a schematic diagram of a real-time diversity storage flow of device operation data in an embodiment of the present application, as shown in fig. 5: after equipment operation data are acquired from a telecom operator network, double writing operation is performed by using the system, and hierarchical storage is automatically completed, and the specific process is as follows:
1. After the collection and analysis of the equipment operation data are completed, the system writes the data into the OpenTSDB and the Redis simultaneously and in real time to perform data double-writing operation.
2. The OpenTSDB can continuously write in sub-tables, store data with the longest storage period and provide cold data storage and query services.
3. Redis will delete the history data periodically according to the configured storage period, and store only recent hot data.
4. After deleting the historical data in the Redis, adding the data with more user inquiry times to the Redis by using an LRU algorithm, and finally realizing the storage of hot data and the elimination of cold data to finish the hierarchical storage of cold and hot data.
FIG. 6 is a flow chart of real-time query of device operational data in an embodiment of the present application, as shown in FIG. 6, the flow includes:
after the application layer obtains the data calling authority from the system, the device operation can be obtained to perform real-time query on the device operation data. After receiving the data call request of the user, the specific program response flow is as follows:
1. and judging whether the searched data belongs to cold data or hot data according to the time range of the query request and the query content.
2.1, if the time range is larger than the set hot data storage period, the query is cold data, and a query interface of the OpenTSDB is called.
2.2, obtaining all sub-table information according to the index and time range transmitted by the user, and initiating concurrent inquiry
And 2.3, calling the modified scan function, and acquiring data meeting the conditions from the HBASE.
And 2.4, judging whether the function calculation requirement exists in the input parameters, and summing, averaging or directly returning the result.
And 3.1, if the time range is smaller than the set hot data storage period, or the query content belongs to the recently common query data, the query is hot data, and a query interface of Redis is called.
And 3.2, acquiring data meeting the conditions from the Redis.
And 3.3, judging whether the function calculation requirement exists in the input parameters, and summing, averaging or directly returning the result.
4. And returning the query result to the user in a unified data format.
Fig. 7 is an apparatus for processing data with multiple databases according to an embodiment of the present application, as shown in fig. 7, the apparatus comprising:
a receiving module 70, configured to receive a data query request of a target object, where the data query request at least carries identification information of data to be queried;
the query module 72 is configured to determine, according to the identification information, a target data type to which the data to be queried belongs, where the data type includes one of the following: hot data, cold data;
The determining module 74 is configured to determine a data query interface corresponding to the target data type, and return the result data queried by the data query interface to the target object in a unified data format, where the target data type corresponds to the data query interface one by one.
In the device, a receiving module 70 is configured to receive a data query request of a target object, where the data query request at least carries identification information of data to be queried; the query module 72 is configured to determine, according to the identification information, a target data type to which the data to be queried belongs, where the data type includes one of the following: hot data, cold data; the determining module 74 is configured to determine a data query interface corresponding to a target data type, and return result data queried by the data query interface to the target object in a uniform data format, where the target data type corresponds to the data query interface one by one, so as to achieve the purpose of classifying and storing cold and hot data based on databases of different types, thereby achieving the technical effects of improving the data processing throughput of the databases, improving the data query efficiency and the response speed, and further solving the technical problems of low data query efficiency and low query response speed caused by poor throughput, low expandability and low query response speed of the traditional relational databases in the related art.
According to another aspect of the embodiment of the present application, there is also provided a nonvolatile storage medium, where the storage medium includes a stored program, and when the program runs, the storage medium is controlled to perform any one of methods for processing data by using multiple databases.
Specifically, the storage medium is configured to store program instructions for the following functions, and implement the following functions:
receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
Alternatively, in the present embodiment, the storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In an exemplary embodiment of the application, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements a method of processing data of a plurality of databases of any of the above.
Optionally, the computer program may, when executed by a processor, implement the steps of:
receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried; determining a target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following: hot data, cold data; determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
According to an embodiment of the present application, there is provided an electronic apparatus including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of processing data in any one of the databases described above.
Optionally, the electronic device may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input device is connected to the processor.
FIG. 8 illustrates a schematic block diagram of an example electronic device 800 that may be used to implement an embodiment of the application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 8, the apparatus 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.
Various components in device 800 are connected to I/O interface 805, including: an input unit 806 such as a keyboard, mouse, etc.; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, or the like. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 801 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a method of processing data in multiple databases. For example, in some embodiments, the method of multi-database processing data may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 800 via ROM 802 and/or communication unit 809. When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the method of processing data for a database described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the method of multi-database processing data in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed technology may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, for example, may be a logic function division, and may be implemented in another manner, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.
Claims (12)
1. A method of processing data in a multi-database, comprising:
receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried;
determining a target data type to which the data to be queried belongs according to the identification information, wherein the data type comprises one of the following: hot data, cold data;
determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
2. The method according to claim 1, wherein determining the target data type to which the data to be queried belongs according to the identification information comprises:
determining a query frequency or a time range of the data to be queried corresponding to the identification information in a historical period, wherein the time range is used for indicating the interval duration between a historical data query request for last receiving the data to be queried and a data query request received at this time;
Determining the data to be queried as the thermal data under the condition that the query frequency is greater than a preset frequency or the time range is less than a preset duration;
and under the condition that the query frequency is smaller than the preset frequency or the time range is larger than the preset duration, determining the data to be queried as the cold data.
3. The method of claim 1, wherein determining a data query interface corresponding to the target data type, and returning data queried by the data query interface to the target object, comprises:
calling a data query interface of an OpenTSDB database under the condition that the target data type is the cold data;
acquiring all sub-table information of the index in the time range from the OpenTSDB database according to the index transmitted by the target object and the selected time range, wherein a scan function in the OpenTSDB database is used for assigning the equipment IP corresponding to the target object to a start time start parameter and an end time end parameter;
judging whether a function calculation requirement exists in parameters input from the target object, carrying out logic operation according to the calculation requirement, and returning a calculation result to the target object.
4. A method according to claim 3, wherein prior to obtaining all sub-table information for the indicator over the time horizon, the method further comprises:
and carrying out query assignment on the device IP by using regular matching of a filter, and changing the query assignment into assignment in a start parameter and an end parameter, wherein the filter is only used for matching port names.
5. The method of claim 1, wherein determining a data query interface corresponding to the target data type, and returning data queried by the data query interface to the target object, comprises:
calling a data query interface of a Redis database under the condition that the target data is the hot data;
judging whether a function calculation requirement exists in parameters input from the target object, carrying out logic operation according to the calculation requirement, and returning a calculation result to a terminal corresponding to the target object.
6. The method of claim 5, wherein prior to invoking the data query interface of the Redis database, the method further comprises:
determining a storage period of the Redis database;
and acquiring historical data written into the Redis database in a historical period, and deleting other data except the storage period in the Redis database.
7. The method of claim 6, wherein after deleting other data in the dis database than the storage period, the method further comprises:
determining data of each object requesting to be queried in the history period, analyzing the data requesting to be queried in the history period based on a page replacement algorithm, screening out new hot data and writing the new hot data into the Redis database.
8. A system for processing data in a plurality of databases, comprising:
the data writing module is provided with a Redis database corresponding to the stored hot data and an OpenTSDB database used for storing cold data;
the data query module is used for receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried, and determines a target data type to which the data to be queried belongs according to the identification information, and the data type comprises one of the following: the hot data, or the cold data; and determining a data query interface corresponding to the target data type, and returning the structural data queried by the data query interface to the target object in a unified data format, wherein the target data type corresponds to the data query interface one by one.
9. The system of claim 8, wherein the scan function in the OpenTSDB database assigns the device IP corresponding to the target object in the start and end parameters.
10. An apparatus for processing data in a plurality of databases, comprising:
the receiving module is used for receiving a data query request of a target object, wherein the data query request at least carries identification information of data to be queried;
the query module is used for determining the target data type of the data to be queried according to the identification information, wherein the data type comprises one of the following data types: hot data, cold data;
and the determining module is used for determining a data query interface corresponding to the target data type, and returning the result data queried by the data query interface to the target object in a uniform data format, wherein the target data type corresponds to the data query interface one by one.
11. A non-volatile storage medium, characterized in that the storage medium comprises a stored program, wherein the program, when run, controls a device in which the storage medium is located to perform the method of processing data of a multi database according to any one of claims 1 to 7.
12. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method of processing data for a multi-database as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310653188.9A CN116680295A (en) | 2023-06-02 | 2023-06-02 | Method, system and device for processing data by multiple databases |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310653188.9A CN116680295A (en) | 2023-06-02 | 2023-06-02 | Method, system and device for processing data by multiple databases |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116680295A true CN116680295A (en) | 2023-09-01 |
Family
ID=87784906
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310653188.9A Pending CN116680295A (en) | 2023-06-02 | 2023-06-02 | Method, system and device for processing data by multiple databases |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116680295A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194298A (en) * | 2023-09-18 | 2023-12-08 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
| CN117194298B (en) * | 2023-09-18 | 2024-05-31 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
-
2023
- 2023-06-02 CN CN202310653188.9A patent/CN116680295A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194298A (en) * | 2023-09-18 | 2023-12-08 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
| CN117194298B (en) * | 2023-09-18 | 2024-05-31 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240946A (en) | The multi-level buffer method and terminal device of data | |
| US20230350894A1 (en) | Distinct value estimation for query planning | |
| CN102446171B (en) | The method and apparatus of keyword quality score is promoted based on the evaluation and test of weighted mean click-through rate | |
| CN112527843B (en) | Data query method, device, terminal equipment and storage medium | |
| US8849745B2 (en) | Decision support methods and apparatus | |
| CN113568938B (en) | Data stream processing method and device, electronic equipment and storage medium | |
| CN108733688B (en) | Data analysis method and device | |
| CN111026761B (en) | Financial data storage system, processing method and device | |
| US20220121638A1 (en) | Index data structures and graphical user interface | |
| CN115344207A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110032578B (en) | Mass data query caching method and device | |
| CN108140022A (en) | Data query method and Database Systems | |
| US7725468B2 (en) | Improving efficiency in processing queries directed to static data sets | |
| CN109726219A (en) | The method and terminal device of data query | |
| CN108897796A (en) | A kind of operation system calls method, storage medium and the server of influxdb database | |
| CN116680295A (en) | Method, system and device for processing data by multiple databases | |
| CN115114289A (en) | Data query method and device and electronic equipment | |
| CN115658728B (en) | Query method, electronic equipment and storage medium | |
| CN115203246A (en) | Linked list query method and device, electronic equipment and storage medium | |
| CN116795893A (en) | Data retrieval method, device, electronic equipment and medium | |
| CN115525659A (en) | Data query method and device, electronic equipment and storage medium | |
| CN115964387A (en) | Data query method and device, distributed database system and medium | |
| CN117609245A (en) | Paging query method, device, equipment and medium | |
| CN115879693A (en) | Task allocation method and device, electronic equipment and storage medium | |
| CN117596298A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |