CN108536823B - Cache design and query method for sensing big data of Internet of things - Google Patents
Cache design and query method for sensing big data of Internet of things Download PDFInfo
- Publication number
- CN108536823B CN108536823B CN201810314923.2A CN201810314923A CN108536823B CN 108536823 B CN108536823 B CN 108536823B CN 201810314923 A CN201810314923 A CN 201810314923A CN 108536823 B CN108536823 B CN 108536823B
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- query
- cache
- perception
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a cache design and query method of sensing big data of the Internet of things, comprising a sensing data storage layer, a cache management layer and a cache management layer, wherein the sensing data storage layer is used for storing original sensing data collected by sensing equipment; the medium perception data layer stores higher value density data with small data size and different granularity size obtained by analyzing and calculating the original perception data, and the data of the medium perception data layer can meet most of inquiry and statistics; the intermediate result cache layer is used for storing intermediate results, reports and query conditions for querying and counting from the visual perception data layer and the original data layer; the mesoscopic perception data layer receives the query request, queries the existing data and selects whether to send the query request to the perception data storage layer; and summarizing and returning the query result step by step. The invention greatly reduces the data amount required by calculation during query, reduces the access times of original sensing data storage, improves the query returning speed and achieves the effect of near real-time query.
Description
Technical Field
The invention belongs to the technical field of computer big data, and relates to a cache design and query method for sensing big data of the Internet of things.
Background
The Internet of things (IoT) is a huge network formed by combining various information sensing devices and the Internet to acquire various required information such as any object or process needing monitoring, connection and interaction in real time, and aims to realize connection between objects, objects and people and connection between all objects and the network, so that identification, management and control are facilitated. The Internet of things industry has the characteristics of long industrial chain and multiple industry groups, and the application range of the Internet of things industry almost covers all industries.
With the rapid development of the internet of things technology and related applications thereof, how to effectively store, query and analyze massive internet of things perception data becomes a key problem to be solved urgently. The sensing data of the internet of things refers to sampling data dynamically acquired by various sensing devices in the internet of things system. Data shows that data on the internet grows 50% annually, doubling every two years, and more than 90% of the data in the world is generated in recent years. According to the monitoring statistics of IDC (international data corporation), it is expected that 35ZB data volume will be owned in total by 2020 globally, which is 20 times the total data volume in 2011. Today, data is the most core resource, and various industries generate massive sensing data every day, for example, in the aspect of intelligent traffic monitoring in the transportation industry, 60 or more than ten thousand devices used for real-time monitoring of GPS position information and roads in beijing are currently in the market, and the traffic sensing data generated every day is about 3 PB.
In order to increase the query speed for the perceptual big data, the industry mainly optimizes from the following five aspects. (1) Column store: column storage models represented by HBase, Hive and Cassandra are receiving more and more attention, and compared with a row storage database, the column storage database can effectively reduce IO (input/output) and improve the compression rate of data when some columns are involved in aggregation query. (2) Lateral expansion of computing power: and a large-scale distributed computing framework represented by MapReduce, Spark and Storm realizes the parallel execution of the query task. (3) Optimizing the performance of the algorithm: in the face of an ultra-large data scale, the algorithm time complexity requirement of a data-intensive calculation task needs to be linear or nearly linear or even sub-linear, so that a statistical analysis algorithm aiming at a traditional database needs to be redesigned, and the optimization of the algorithm performance faces a considerable challenge in the environment of the internet of things. (4) A memory database: the memory database represented by Redis, RAMCLOUD and Memcached realizes data storage by using a memory, and is used as a data cache to improve throughput and reduce delay. (5) And (3) optimizing the index: and constructing indexes for the stored data to realize high-performance data retrieval service.
However, the existing methods of internet of things, big data processing technology or products focusing on signal transmission and original sensing data storage are not capable of timely and effectively analyzing and utilizing the sensing big data with high hidden value. Due to the massive characteristics of the data, the requirements of near real-time performance are still difficult to achieve by optimizing the original sensing data. In the aspect of calculation, even if the time complexity of the algorithm meets the linear requirement, the speed still cannot reach the second-level query under the condition of large target data volume. In the aspect of caching, the traditional method improves the query speed by enabling a memory to store common query data, but when a large range of data is queried, a large amount of original sensing data must be read from a data storage system. In the aspect of indexing, the scanning range is reduced by establishing an index, but the mass characteristic of original data determines that the data volume to be scanned is still huge, and particularly relates to large-range query calculation. Therefore, the massive characteristics of the sensing data of the internet of things are still difficult to deal with through the optimization, and the key technical problems need to be further researched for realizing near-real-time query of the sensing data of the internet of things.
Disclosure of Invention
Aiming at the key problems which are not solved, the invention provides a cache design and an inquiry mechanism of sensing big data of the Internet of things, which realize the near real-time inquiry of the sensing big data and achieve the effect of inquiring and accelerating mainly by adding an intermediate result cache layer and a middle viewing sensing data layer between a service system and an original sensing data storage system.
The system structure of the invention is shown in figure 1, the system for sensing big data of the Internet of things is divided into 4 layers, the first layer is a sensing data storage layer and is responsible for storing original sensing data, and the data are massive detailed information collected by sensing equipment and have microscopic characteristics; the second layer is a middle-view perception data layer, which is used for analyzing and calculating in advance on the basis of original perception data and converting the original perception data with low value density and large data volume into data with high value density and small data volume. Each piece of high-value density data is obtained by analyzing and calculating a plurality of pieces of original sensing data; the third layer is an intermediate result cache layer which caches intermediate result data obtained by inquiring and calculating from the observation sensing data layer and common statistical report data; the fourth layer is a business system layer, provides results or reports required by inquiry or statistics and has macroscopic characteristics. Wherein, the central perception data layer is focused, namely, the data with different granularities obtained by analyzing and calculating on the basis of storing the original perception data. By means of the data, most of the near real-time query requirements are met.
In order to achieve the purpose, the invention adopts the following technical scheme:
a cache design and query method for sensing big data of the Internet of things comprises the following steps:
step 1, converting the original perception data into the mesoscopic perception data and storing the mesoscopic perception data.
And 2, converting the fine-grained mesoscopic perception data into coarse-grained mesoscopic perception data and storing the coarse-grained mesoscopic perception data.
And step 3, periodically executing the operations of the step 1 and the step 2.
And 4, customizing the analysis mode and the execution rule of the query engine in the visual perception data layer on various queries and the returned data template.
And 5, maintaining data and configuration in the middle perception data layer, wherein the data and configuration comprise deleting and recalculating the middle perception data.
And 6, analyzing and calculating the data of the mesoscopic perception layer, and caching the result in a middle result caching layer.
And 7, periodically executing the step 6.
And 8, caching the query or calculation results meeting the caching conditions in an intermediate result caching layer.
And 9, customizing the analysis mode and the execution rule of the query engine in the intermediate result cache layer on various queries and the returned data template.
And step 10, maintaining data and configuration in the intermediate result cache layer, deleting or recalculating cache data and the like.
The process of converting the original sensing data into the mesoscopic sensing data in the steps 1 and 3 is as follows:
aiming at the existing original perception data, performing one-time traversal calculation for the first time to obtain the perception data in fine granularity; and then, a periodic timing task is formulated, and the task performs calculation and statistics on newly added original sensor data and other original data each time to obtain fine-grained mesoscopic perception data which are stored in a mesoscopic perception data layer.
The process of converting the fine-grained mesoscopic perception data into coarse-grained mesoscopic perception data in the step 2 and the step 3 is as follows:
and aiming at the visual perception data in the fine granularity, a plurality of periodic timing tasks are formulated, each task analyzes and calculates the newly added visual perception data in the fine granularity to obtain the visual perception data in the coarse granularity, and the visual perception data is stored in a visual perception data layer to obtain the visual perception data of different granularities and different classes for meeting the business requirements.
The intermediate perception data storage in step 1 and step 2 is described as follows:
and the medium-looking sensing data layer selects the Druid as a storage system, and the metadata is stored in the relational database mysql. The Druid is a distributed data processing system supporting real-time multidimensional OLAP analysis. The Druid is selected as a storage system of the central perception data layer, and the Druid supports high-speed data real-time intake processing; real-time and flexible multi-dimensional data analysis query is supported; pre-polymerization uptake and polymerization analysis of the data according to the time stamp is supported. The internet of things perception data has a time sequence characteristic, flexible and rapid multi-dimensional OLAP analysis is carried out on the basis of the mesoscopic perception data layer, and therefore the Druid is selected as the storage of the mesoscopic perception data layer.
The caching process in the step 6 means that analysis and calculation are performed on the intermediate perception data layer at one time according to configuration for the first time, and then the intermediate perception data layer is cached.
The tasks periodically executed in step 7 are described as follows:
and formulating a periodic timing task, analyzing and calculating the newly added perception data in each granularity by the task, and updating corresponding data in the intermediate result cache layer according to a calculation result.
In the step 8, the cache data is cached according to the configuration in the service query process, and if the cache replacement policy of the part is not configured, the LRU is adopted by default.
The storage of the intermediate result cache layer in steps 6 and 8 is described as follows:
the bottom layer cache technology uses Redis, a Redis Cluster architecture is selected, and the Redis Cluster has the following advantages: the data fragmentation function is supported, and data can be distributed to different instances; high availability of service, automatic transfer of failure, avoiding single point failure to the greatest extent; online horizontal expansion capability, online addition of nodes, data transfer, and the like; a centerless architecture, degrees of each node, and the like; the complexity of the original data slicing scheme is reduced, and hardware resources are saved; less system bottleneck, client direct connection mode and the like. The caching requirement as an intermediate result caching layer is met.
In the step 4, the step 5, the step 9 and the step 10, the configuration information of the mesoscopic perception data layer and the configuration information of the intermediate result cache layer are stored by adopting a relational database mysql and a file, and the operation modes of the mesoscopic perception data layer and the intermediate result cache layer comprise a command line client mode and a web client mode. The command line client is used for ssh or accessing on a deployment machine; the remote web client uses a browser for access.
When the method executes the query, as shown in fig. 2 and fig. 3, the query flow is described as follows:
and the business system client sends out a query request. The query engine of the intermediate result cache layer directly queries and returns the data if the cache completely meets the query condition according to the comparison of the existing cache data; if no matching data exists in the cache, directly issuing the query; if the two parts meet the requirement, the query is segmented, the query in the intermediate result cache layer is asynchronously executed, the query request is issued to the intermediate perception data layer, and the result is merged after the result is returned. And the query engine of the middle perception data layer receives the query request, compares the existing data in the middle perception data layer, judges whether the query request is met, and similarly, if the query request is met, the query is carried out in the middle perception data layer, if no matched data exists, the query request is directly issued, if part of the query request is met, the query is segmented, the query in the middle perception data layer is asynchronously executed, the query is issued to the bottom original perception data storage system, and after the result is returned, the result is integrated and returned to the middle result cache layer. In the process, the query result data which accords with the configuration rule is asynchronously cached to the intermediate result, so that the next query is facilitated.
In conclusion, the method and the system can ensure that most queries of the service system can obtain results from the intermediate result cache layer and the intermediate perception data layer in time, greatly reduce the access times to the original perception data storage system, reduce repeated calculation, improve the query and statistics speed, and meet the requirement of near-real-time query.
Drawings
FIG. 1: sensing big data caching and inquiring a system architecture diagram by the Internet of things;
FIG. 2: the functions and data flow direction of the middle observation sensing data layer and the middle result cache layer are simplified;
FIG. 3: sensing big data caching and inquiring system inquiry flow chart.
Detailed Description
The following detailed explanation and explanation of the system implementation are made in conjunction with the accompanying drawings, and the implementation and operation procedures are given on the premise of the technical scheme of the system, but the protection scope of the system is not limited to the implementation.
As shown in fig. 1, the system is a caching and querying system between an internet of things raw sensing data storage system and a service system, and includes: an intermediate result caching layer and an intermediate look-and-feel data layer. The intermediate result cache data layer comprises an inquiry engine module, a configuration management module, a timing task execution module and a data management module. The intermediate result cache layer is used for caching query statistics on the intermediate perception data layer to obtain a query result or a common report, and comprises a query engine module, a configuration management module, a timing task execution module and a data management module, and is used for storing data with finer granularity obtained by statistics and calculation on the basis of original perception data and obtaining data with coarser granularity obtained by statistics and calculation of the data with finer granularity.
The principle that the system can improve the query and statistic speed of the service system is as follows: the system analyzes and calculates in advance on the basis of original perception data, converts the original perception data with low value density and large data volume into data with high value density and small data volume, uses the data as fine-grained intermediate perception data, calculates other intermediate perception data with different granularities according to the fine-grained intermediate perception data, and stores the data in an intermediate perception data layer; the system caches the intermediate result data obtained by inquiring and calculating from the visual perception data layer and the common statistical report data in an intermediate result cache layer. The data of different granularities in the data layer and the data in the intermediate result cache layer are observed and sensed in the system, so that most of business queries can be met, and the requirement of near-real-time query speed is met.
The implementation steps of the system are added between a service system and an original sensing data storage system as follows:
(1) operating the initial configuration and initial task of metadata in a configuration management module of the mesoscopic perception layer: and analyzing and calculating the original sensing data according to the characteristics of the original sensing data and the query requirement of the service system to obtain fine-grained data of the medium-looking sensing data layer. For example, a taxi keeps traveling on a certain road for a certain period of time, and the original sensing data record is continuously generated: the method comprises the following steps of calculating information such as vehicle id, timestamp, position information, speed, passenger carrying and the like to a plurality of pieces of perception data in a period of time to obtain a record in a middle-view perception data layer: vehicle id, time range (1-10 minutes), location range, speed range, location vector function, speed vector function, passenger load, maximum speed value, minimum speed value, average speed value, distance, etc. The vector function is used for describing in the medium-view sensing data layer, and the state of the monitored object is depicted for a period of time, so that the data accuracy is guaranteed, the data quantity of the medium-view sensing data layer is greatly reduced, and in addition, common calculations such as distance, maximum and minimum values and the like are directly stored in the medium-view sensing data layer, and the recalculation is avoided.
(2) Operating a task module in a configuration management module of the mesoscopic perception layer, and calculating fine-grained mesoscopic perception data to obtain different coarse-grained mesoscopic perception data, such as the fine-grained data in the step (1): vehicle id, time range (1-10 minutes), location range, speed range, location vector function, speed vector function, passenger capacity, maximum speed value, minimum speed value, average speed value, distance, etc., and coarse-grained perception data obtained by calculation: vehicle id, time range (by day, by week, by month, etc.), location range, speed range, passenger load, maximum speed value, minimum speed value, average speed value, distance, etc. The sensing data with the coarse granularity has information loss relative to the sensing data with the fine granularity, but can meet the inquiry and statistics requirements of the coarse granularity.
(3) Configuring and executing an initialization task and a periodic timing task in a mesoscopic perception data layer, wherein the initialization task is to analyze and analyze original perception data to obtain mesoscopic perception data with fine granularity and sequentially obtain mesoscopic perception data with coarse granularity; and the periodic timing task is used for calculating the increment original sensing data and the increment intermediate sensing data to generate and store corresponding intermediate sensing data.
(4) And configuring analysis modes, execution rules, returned data templates, integration rules and the like of the viewing perception data layer query engine for various queries.
(5) And operating a configuration management module in the intermediate result cache layer, and configuring an initialization task, a periodic timing task, cache rule configuration, query execution rules and the like. The initialization task is to cache report data with macroscopic characteristics obtained by counting and calculating in the observation and perception data layer; the periodic timing task is to calculate incremental data in the central perception data layer and update data in the cache in time; the cache rule is a self-defined cache condition, the query result meeting the cache condition is asynchronously cached during query, and cache replacement strategies of various queries are self-defined or selected, if the default LRU strategy is not configured. The query execution rule refers to customizing the analysis mode and the execution rule of the query engine in the intermediate result cache layer to each query, a returned data template and the like.
(6) And starting various services of the intermediate observation and perception data layer and the intermediate result caching layer.
The query request is sent from the business system, and the steps of performing the query in the system are described as follows (see fig. 3):
(1) a business system client sends out a query request, a query engine of an intermediate result cache layer receives the query request and analyzes the query, a query rule is matched, whether the cache data meet the query condition or not is judged according to the query rule and the existing cache data, and if the cache data meet the query condition, the query is executed in the cache; if the cache does not have matched data, the query is directly issued, if the cache matches partial query conditions, the query needs to be segmented, the query in the cache is asynchronously executed, the query is issued to the mesoscopic perception data layer, and after the result is returned, the query result is integrated and returned to the service system.
(2) The query engine of the mesoscopic perception data layer receives and analyzes the query request, matches the query rule, checks whether the query request needs to be segmented or directly issues the query in the mesoscopic perception data layer according to the query rule, and selects the data with the corresponding granularity for query or calculation if the data of the mesoscopic perception data layer meets the query condition; if the mesoscopic perception data layer does not have data meeting the conditions, directly issuing the query; and if the partial results meet the requirements, segmenting the query, asynchronously executing the query in the layer and issuing the query to the bottom original sensing data storage system, and after the result is returned, integrating the query result and returning the result to the intermediate result cache layer.
(3) And after receiving the query result returned by the original perception data storage system, the intermediate perception data layer integrates the result according to the rule selection template and returns the result to the intermediate result data layer in real time, and the intermediate result data layer integrates the result and returns the result to the service system in real time. In the process, the query result and the query condition which accord with the configuration rule are asynchronously cached to the intermediate result cache layer or stored to the intermediate perception data layer, so that the next query is facilitated.
Through verification, the method can greatly reduce the access times to the original sensing data storage system, reduce repeated calculation, improve the speed of inquiry and statistics and meet the requirement of near-real-time inquiry.
Examples
A cache and query system for massive perceptual data, which is designed for near real-time query and statistics on the massive perceptual data, the system comprising:
and the perception data storage layer is responsible for storing original perception data, and the perception data are detailed information collected by a perception device.
The medium-view perception data layer is composed of a large-capacity distributed large-data storage system, original perception data with low value density and large data volume are converted into medium-view perception data with high value density and small data volume and stored in the medium-view perception data layer, the medium-view perception data layer contains data with different granularities in all dimensions, for example, the different granularities in the time dimension can be divided into hourly, daily, weekly and monthly.
And the intermediate result caching layer consists of a high-capacity distributed caching system and caches the common statistical report data and intermediate result data which is obtained by inquiring and calculating from the visual perception data layer and accords with the configuration rule.
And the business system layer provides a result or report required by query or statistics for the user.
The mass sensing data caching and inquiring system is characterized in that the medium-view sensing data layer mainly comprises an inquiring engine module, a data management module, a task management module and a configuration management module. The medium-view sensing data layer stores view sensing data in different granularities, and the data sources are as follows: (1) the method comprises the steps that original sensing data are analyzed and calculated at one time during initialization; (2) the method comprises the steps that an incremental part in an original sensing data layer is obtained through analysis and calculation periodically; (3) the incremental data in the visual perception data layer are periodically analyzed and calculated.
The mass sensing data caching and inquiring system is characterized in that the intermediate result caching layer mainly comprises an inquiring engine module, a data management module, a task management module and a configuration management module. The data sources in the intermediate result cache layer are: (1) the data in the central perception data layer is obtained by analyzing and calculating at one time during initialization; (2) the incremental data in the central perception data layer are periodically analyzed and calculated to obtain the incremental data; (3) and caching the intermediate query result of the matching rule during query to obtain the query result.
The method for querying the mass sensing data cache and the query system comprises the following steps:
step 1, a business system layer sends out a query request, an intermediate result cache layer query engine receives the query request, analyzes and matches the query rule, judges whether the data in the intermediate result cache layer meets the query condition according to the query rule and the existing cache data, and executes query in the intermediate result cache layer if the data meets the query condition; if no matching data exists in the cache, directly issuing the query; if the cache matches partial query conditions, the query needs to be segmented, the query in the cache is executed asynchronously, and the query is issued to the mesoscopic perception data layer.
Step 2, the intermediate result cache layer issues query requests, a query engine of the intermediate perception data layer receives the query requests, analyzes and matches query rules, checks whether the query requests need to be segmented or directly issues queries in the intermediate perception data layer according to the query rules, and selects data of corresponding granularity for query and calculation if the data of the layer meets query conditions; if the data in the medium perception data layer does not have data meeting the conditions, directly issuing the query; and if the partial query is segmented, asynchronously executing the query in the layer and issuing the query to the original perception data storage layer.
And 3, the intermediate perception data layer collects the query result from the original data storage layer and the query calculation result from the layer, packages the data according to the corresponding query result template, and returns the data to the intermediate result cache layer in real time.
And 4, the intermediate result cache layer collects the query result from the intermediate perception data layer and the query calculation result from the local layer, generates the data to be cached and the query result required by the service system layer according to the corresponding query result template, and returns the result to the service system layer in real time.
By using the method, only the observation sensing data in each granularity of small data volume needs to be additionally stored, and the common intermediate query result and the statistical report are cached.
Claims (10)
1. A cache design and query method of sensing big data of the Internet of things is characterized in that a system of sensing big data of the Internet of things is divided into 4 layers, the first layer is a sensing data storage layer and is responsible for storing original sensing data, and the data are massive detailed information collected by sensing equipment and have microscopic characteristics; the second layer is a middle-view perception data layer, which is used for analyzing and calculating in advance on the basis of original perception data and converting the original perception data with low value density and large data volume into data with high value density and small data volume; each piece of high-value density data is obtained by analyzing and calculating a plurality of pieces of original sensing data; the third layer is an intermediate result cache layer which caches intermediate result data obtained by inquiring and calculating from the observation sensing data layer and common statistical report data; the fourth layer is a business system layer, provides results or reports required by query or statistics, and has macroscopic characteristics; wherein, the observation perception data layer in the key focus, namely, the data with different granularities obtained by analyzing and calculating on the basis of storing the original perception data; most of near real-time query requirements are met through the data;
the method is characterized in that: the method comprises the following steps:
step 1, converting original perception data into mesoscopic perception data and storing the mesoscopic perception data;
step 2, converting the fine-grained mesoscopic perception data into coarse-grained mesoscopic perception data and storing the coarse-grained mesoscopic perception data;
step 3, periodically executing the operations of the step 1 and the step 2;
step 4, customizing the analysis mode and the execution rule of the query engine in the middle perception data layer to various queries and the returned data template;
step 5, maintaining data and configuration in the middle perception data layer, including deleting and recalculating middle perception data;
step 6, analyzing and calculating results of the mesoscopic perception layer data, and caching the results in an intermediate result caching layer;
step 7, periodically executing step 6;
step 8, caching the query or calculation results meeting the caching conditions in an intermediate result caching layer;
step 9, customizing the analysis mode and the execution rule of the query engine in the intermediate result cache layer to various queries and the returned data template;
and step 10, maintaining data and configuration in the intermediate result cache layer, and deleting or recalculating cache data.
2. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: the process of converting the original sensing data into the mesoscopic sensing data in the steps 1 and 3 is as follows:
aiming at the existing original perception data, performing one-time traversal calculation for the first time to obtain the perception data in fine granularity; and then, a periodic timing task is formulated, and the task performs calculation and statistics on newly added original sensor data and other original data each time to obtain fine-grained mesoscopic perception data which are stored in a mesoscopic perception data layer.
3. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: the process of converting the fine-grained mesoscopic perception data into coarse-grained mesoscopic perception data in the step 2 and the step 3 is as follows:
and aiming at the visual perception data in the fine granularity, a plurality of periodic timing tasks are formulated, each task analyzes and calculates the newly added visual perception data in the fine granularity to obtain the visual perception data in the coarse granularity, and the visual perception data is stored in a visual perception data layer to obtain the visual perception data of different granularities and different classes for meeting the business requirements.
4. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: the intermediate perception data storage in step 1 and step 2 is described as follows:
the middle-view sensing data layer selects a Druid as a storage system, and metadata is stored in a relational database mysql; wherein the Druid is a distributed data processing system supporting real-time multidimensional OLAP analysis; the Druid is selected as a storage system of the central perception data layer, and the Druid supports high-speed data real-time intake processing; real-time and flexible multi-dimensional data analysis query is supported; support pre-polymerization uptake and polymerization analysis of data according to time stamps; the internet of things perception data has a time sequence characteristic, flexible and rapid multi-dimensional OLAP analysis is carried out on the basis of the mesoscopic perception data layer, and therefore the Druid is selected as the storage of the mesoscopic perception data layer.
5. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: the caching process in the step 6 means that analysis and calculation are performed on the visual perception data layer at one time according to configuration for the first time, and then the cache is stored in the middle result caching layer.
6. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: the tasks periodically executed in step 7 are described as follows:
and formulating a periodic timing task, analyzing and calculating the newly added perception data in each granularity by the task, and updating corresponding data in the intermediate result cache layer according to a calculation result.
7. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: in the step 8, the cache data is cached according to the configuration in the service query process, and if the cache replacement policy of the part is not configured, the LRU is adopted by default.
8. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: the storage of the intermediate result cache layer in steps 6 and 8 is described as follows:
the bottom layer cache technology uses Redis, a Redis Cluster architecture is selected, and the Redis Cluster has the following advantages: the data fragmentation function is supported, and data can be distributed to different instances; high availability of service, automatic transfer of failure, avoiding single point failure to the greatest extent; the capacity is expanded horizontally on line, nodes are added on line, and data are transferred; no central architecture, each node degree; the complexity of the original data slicing scheme is reduced, and hardware resources are saved; the system bottleneck is less, and the client side is in a direct connection mode; the caching requirement as an intermediate result caching layer is met.
9. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: in the step 4, the step 5, the step 9 and the step 10, the configuration information of the mesoscopic perception data layer and the configuration information of the intermediate result cache layer are stored by adopting a relational database mysql and a file, and the operation modes of the mesoscopic perception data layer and the intermediate result cache layer comprise a command line client and a web client; the command line client is used for ssh or accessing on a deployment machine; the remote web client uses a browser for access.
10. The cache design and query method for sensing big data of the internet of things according to claim 1, wherein the cache design and query method comprises the following steps: when the method executes the query, the query flow is described as follows:
a business system client sends out a query request; the query engine of the intermediate result cache layer directly queries and returns the data if the cache completely meets the query condition according to the comparison of the existing cache data; if no matching data exists in the cache, directly issuing the query; if the partial results meet the requirements, the query is segmented, the query in the intermediate result cache layer is asynchronously executed, the query request is issued to the intermediate perception data layer, and the results are merged after the results are returned; the query engine of the middle perception data layer receives the query request, compares the existing data in the middle perception data layer, judges whether the query request is satisfied, and similarly, if so, queries in the middle perception data layer, if not, directly issues the query request, if partially satisfied, segments the query, asynchronously executes the query in the middle perception data layer and issues the query to the bottom original perception data storage system, and after the result is returned, integrates the result and returns to the middle result cache layer; in the process, the query result data which accords with the configuration rule is asynchronously cached to the intermediate result, so that the next query is facilitated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810314923.2A CN108536823B (en) | 2018-04-10 | 2018-04-10 | Cache design and query method for sensing big data of Internet of things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810314923.2A CN108536823B (en) | 2018-04-10 | 2018-04-10 | Cache design and query method for sensing big data of Internet of things |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108536823A CN108536823A (en) | 2018-09-14 |
| CN108536823B true CN108536823B (en) | 2022-02-15 |
Family
ID=63480637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810314923.2A Active CN108536823B (en) | 2018-04-10 | 2018-04-10 | Cache design and query method for sensing big data of Internet of things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108536823B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457341B (en) * | 2019-07-03 | 2024-05-07 | 平安科技（深圳）有限公司 | Data aggregation method, device, computer equipment and storage medium |
| CN111966727A (en) * | 2020-08-12 | 2020-11-20 | 北京海致网聚信息技术有限公司 | Spark and Hive based distributed OLAP (on-line analytical processing) ad hoc query method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101957927A (en) * | 2010-11-12 | 2011-01-26 | 福州联迅信息科技有限公司 | Middleware architecture of Internet of things and SOA architecture-based middleware of Internet of things |
| CN103138965A (en) * | 2011-11-28 | 2013-06-05 | 中国电信股份有限公司 | Method, device and system for inquiring device state of IOT (Internet of Things) |
| CN103810441A (en) * | 2014-01-28 | 2014-05-21 | 浙江大学 | Multi-granularity remote sensing data access method based on rules |
| CN104008212A (en) * | 2014-06-23 | 2014-08-27 | 中国科学院重庆绿色智能技术研究院 | Method for storing IOT time series data related to geographical location information |
| CN105306095A (en) * | 2015-09-25 | 2016-02-03 | 中国人民解放军国防科学技术大学 | Method and system for rapidly capturing relay satellite measurement and control system signal |
| CN106227899A (en) * | 2016-08-31 | 2016-12-14 | 北京京航计算通讯研究所 | The storage of the big data of a kind of internet of things oriented and querying method |
| CN106528448A (en) * | 2016-10-11 | 2017-03-22 | 杭州数强网络科技有限公司 | Distributed caching mechanism for multi-source heterogeneous electronic commerce big data |
| CN107391744A (en) * | 2017-08-10 | 2017-11-24 | 东软集团股份有限公司 | Data storage, read method, device and its equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8275815B2 (en) * | 2008-08-25 | 2012-09-25 | International Business Machines Corporation | Transactional processing for clustered file systems |
| US8285969B2 (en) * | 2009-09-02 | 2012-10-09 | International Business Machines Corporation | Reducing broadcasts in multiprocessors |
| US8762425B2 (en) * | 2010-10-18 | 2014-06-24 | Hewlett-Packard Development Company, L.P. | Managing a data structure |
| US11321268B2 (en) * | 2014-10-31 | 2022-05-03 | Texas Instruments Incorporated | Multicore bus architecture with wire reduction and physical congestion minimization via shared transaction channels |
-
2018
- 2018-04-10 CN CN201810314923.2A patent/CN108536823B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101957927A (en) * | 2010-11-12 | 2011-01-26 | 福州联迅信息科技有限公司 | Middleware architecture of Internet of things and SOA architecture-based middleware of Internet of things |
| CN103138965A (en) * | 2011-11-28 | 2013-06-05 | 中国电信股份有限公司 | Method, device and system for inquiring device state of IOT (Internet of Things) |
| CN103810441A (en) * | 2014-01-28 | 2014-05-21 | 浙江大学 | Multi-granularity remote sensing data access method based on rules |
| CN104008212A (en) * | 2014-06-23 | 2014-08-27 | 中国科学院重庆绿色智能技术研究院 | Method for storing IOT time series data related to geographical location information |
| CN105306095A (en) * | 2015-09-25 | 2016-02-03 | 中国人民解放军国防科学技术大学 | Method and system for rapidly capturing relay satellite measurement and control system signal |
| CN106227899A (en) * | 2016-08-31 | 2016-12-14 | 北京京航计算通讯研究所 | The storage of the big data of a kind of internet of things oriented and querying method |
| CN106528448A (en) * | 2016-10-11 | 2017-03-22 | 杭州数强网络科技有限公司 | Distributed caching mechanism for multi-source heterogeneous electronic commerce big data |
| CN107391744A (en) * | 2017-08-10 | 2017-11-24 | 东软集团股份有限公司 | Data storage, read method, device and its equipment |
Non-Patent Citations (7)
| Title |
|---|
| Intelligent Greenhouse Clean Energy Control Integrating Multi-Granularity Internet of Things;Jing Shu 等;《 2018 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS)》;20180409;第568-571页 * |
| Text-Based Event Temporal Resolution and Reasoning for Information Analytics in Big Data;Junsheng Zhang 等;《 2015 International Conference on Identification, Information, and Knowledge in the Internet of Things (IIKI)》;20160310;第78-81页 * |
| 大数据安全保障关键技术分析综述;王丹 等;《北京工业大学学报》;20170316;第43卷(第3期);第335-349页 * |
| 物联网大数据存储与管理技术研究;郝行军;《中国博士学位论文全文数据库 信息科技辑》;20170915(第09(2017)期);第I138-50页 * |
| 物联网异构物品解析与信息发现的研究与设计;朱珠;《中国优秀硕士学位论文全文数据库 信息科技辑》;20160115(第01(2016)期);第I138-772页 * |
| 物联网感知大数据分层存储和查询技术研究;蔺春华;《中国优秀硕士学位论文全文数据库 信息科技辑》;20190515(第05(2019)期);第I136-352页 * |
| 面向物联网的多元标识映射模型;王平泉 等;《中国科学:信息科学》;20131015;第43卷(第10期);第1244-1264页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108536823A (en) | 2018-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20160246829A1 (en) | Managing time series databases | |
| Mokbel et al. | SOLE: scalable on-line execution of continuous queries on spatio-temporal data streams | |
| EP2350868B1 (en) | Efficient large-scale filtering and/or sorting for querying of column based data encoded structures | |
| US20140156636A1 (en) | Dynamic parallel aggregation with hybrid batch flushing | |
| CN106649687B (en) | Big data online analysis processing method and device | |
| CN100416560C (en) | Method and apparatus for clustered evolving data flow through on-line and off-line assembly | |
| JP2017518584A (en) | Method for flash optimized data layout, apparatus for flash optimized storage, and computer program | |
| CN106528847A (en) | Multi-dimensional processing method and system for massive data | |
| Havers et al. | Driven: a framework for efficient data retrieval and clustering in vehicular networks | |
| CN108536823B (en) | Cache design and query method for sensing big data of Internet of things | |
| US20220019764A1 (en) | Method and device for classifying face image, electronic device and storage medium | |
| Cuzzocrea et al. | Approximate range–sum query answering on data cubes with probabilistic guarantees | |
| Havers et al. | DRIVEN: A framework for efficient Data Retrieval and clustering in Vehicular Networks | |
| Tian et al. | A survey of spatio-temporal big data indexing methods in distributed environment | |
| AU2020101071A4 (en) | A Parallel Association Mining Algorithm for Analyzing Passenger Travel Characteristics | |
| Gaurav et al. | An outline on big data and big data analytics | |
| Colmenares et al. | A single-node datastore for high-velocity multidimensional sensor data | |
| Jensen et al. | Time series management systems: a 2022 survey | |
| Hua et al. | SmartStore: A new metadata organization paradigm with metadata semantic-awareness for next-generation file systems | |
| Li et al. | SP-phoenix: a massive spatial point data management system based on phoenix | |
| Zhang et al. | Big data storage technology suitable for the operation and maintenance of new generation power grid dispatching control system operation | |
| Song et al. | A distributed self-adaption cube building model based on query log | |
| Liao et al. | Optimizing data query performance of Bi-cluster for large-scale scientific data in supercomputers | |
| Dia et al. | Fast SPARQL join processing between distributed streams and stored RDF graphs using bloom filters | |
| CN112286995B (en) | Data analysis method, device, server, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |