CN111935124A - Multi-source heterogeneous data compression method applied to smart city - Google Patents
Multi-source heterogeneous data compression method applied to smart city Download PDFInfo
- Publication number
- CN111935124A CN111935124A CN202010771901.6A CN202010771901A CN111935124A CN 111935124 A CN111935124 A CN 111935124A CN 202010771901 A CN202010771901 A CN 202010771901A CN 111935124 A CN111935124 A CN 111935124A
- Authority
- CN
- China
- Prior art keywords
- data
- dts
- sampling rate
- sampling
- different
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computer Security & Cryptography (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses multisource heterogeneous data compression method applied to smart city, including: the data transmission method comprises the steps that a data transmission unit DTS acquires data uploaded by a plurality of terminal devices, wherein the data types comprise IoT data, GIS data and BIM data, the data comprise time attributes and region attributes, the terminal devices are networked by adopting a multi-source heterogeneous network, and the DTS is located on a data aggregation layer; the DTS carries out grouping and data cleaning on the acquired data according to different data types to generate a matrix type original data group; the DTS samples the original data group according to a preset sampling model to obtain a plurality of sample data of the original data group, wherein the sample data amount is lower than the data amount of the matrix type original data group; and the DTS performs data splicing on the plurality of sample data sets according to a preset format to form a compressed data set, and sends the compressed data set to the core layer.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a multi-source heterogeneous data compression method applied to a smart city.
Background
The City Information Modeling (CIM) is a digital expression and description of various entity targets and space-time states of the ground, underground, indoor and outdoor of a City, reflects City planning, construction, development and operation, and can be used for City planning decision, City construction, City management and other works.
CIM is a concept with a large span, and relates to industries including various industries such as planning, China and soil, traffic, water conservancy, security, civil air defense, environmental protection, cultural relic protection, energy and gas and the like and all fields related to smart cities.
At present, the CIM is lack of systematic intensive research at home and abroad, and according to the view point in the article of the City information model related technology development review under the intelligent city background, the primary analysis of the basic characteristics of the CIM from three words forming the CIM can be tried: firstly, City, wherein the CIM is to cover the City scale, the City can be instantiated as a City or a City area, a garden, a community, a courtyard, etc., but the description capability of the CIM on the modeling object should be at City level; secondly, Information is Information, Information contained in the CIM covers various space and time dimensions and can support various urban applications, and the Information in the CIM can describe various physical or human entities of the city and has the characteristics of multi-tense, multi-type, multi-granularity level, multi-source and the like; finally, Modeling, i.e., CIM, organizes, simulates, analyzes, and expresses the above information as needed based on certain rules and methods, and further, aggregates intelligence by fusing, mining, and abstracting new knowledge.
From the current development of CIM, CIM is mainly closely related to techniques such as BIM (Building Information Modeling), GIS (Geographic Information System), IOT (Internet of Things), and the like, and meanwhile, it is inevitably required to apply to new-generation Information techniques such as cloud computing and big data.
In the prior art, for multisource heterogeneous data of a smart city, the huge data volume cannot be effectively stored, the multisource heterogeneous data must be compressed, and the key for ensuring that the smart city data can be effectively utilized is to compress and store mass data in different categories. However, there is no effective data compression mechanism in the prior art, which can perform data compression for different time, regions and service types, resulting in large amount of stored data and low compression efficiency.
Disclosure of Invention
The embodiment of the application provides a multisource heterogeneous data compression method applied to a smart city, and the method is used for solving the problem of low multisource heterogeneous data compression efficiency in a smart city scene in the prior art.
The embodiment of the invention provides a multi-source heterogeneous data compression method applied to a smart city, which comprises the following steps:
the data transmission method comprises the steps that a data transmission unit DTS acquires data uploaded by a plurality of terminal devices, wherein the data types comprise IoT data, GIS data and BIM data, the data comprise time attributes and region attributes, the terminal devices are networked by adopting a multi-source heterogeneous network, and the DTS is located on a data aggregation layer;
the DTS carries out grouping and data cleaning on the acquired data according to different data types to generate a matrix type original data group;
the DTS samples the original data group according to a preset sampling model to obtain a plurality of sample data of the original data group, wherein the sample data amount is lower than the data amount of the matrix type original data group;
and the DTS performs data splicing on the plurality of sample data sets according to a preset format to form a compressed data set, and sends the compressed data set to the core layer.
Optionally, the DTS performs sampling on the original data set according to a preset sampling model, and acquires a plurality of sample data of the original data set, including:
the DTS sorts the original data groups according to time sequence, acquires historical change rates of data in different time periods based on corresponding relations between different time sequences and data sizes under the condition of the same region and the same data type, sets a first sampling rate of the data which dynamically changes along with time based on the historical change rates, wherein the first sampling rate is in direct proportion to the data change rate,
and the number of the first and second groups,
the DTS sets a second sampling rate of data based on the region attributes, the second sampling rate having a one-to-one correspondence with the different region attributes,
and the number of the first and second groups,
the DTS sets a dynamically varying third sampling rate based on the different data types, the third sampling rate corresponding one-to-one to the data types,
and the DTS samples the original data group in sequence based on the first sampling rate, the second sampling rate and the third sampling rate, and acquires a plurality of sampled data groups.
Optionally, the DTS performs sampling on the original data group according to a preset sampling model, including:
and the DTS sets a self-adaptive sampling rate based on the different data types and the frequency of the data reporting heartbeat packet, and samples the original data group through the self-adaptive sampling rate.
Optionally, after the data assembling is performed on the plurality of sample data sets according to the preset format and before the compressed data set is formed, the method further includes:
carrying out binary conversion on the assembled data group;
and combining repeated data into the same data bit in the binary data bits, and setting a secondary digit for representing the repeated times of the data bits.
Optionally, the DTS performs sampling on the original data group according to a preset sampling model, including:
predicting the data change rate of different data types at different time periods by adopting a machine learning algorithm;
setting an adaptive sampling rate based on the predicted data change rate, the adaptive sampling rate corresponding to the data change rate one to one;
and sampling the original data group by the self-adaptive sampling rate, generating different time identifications, and inserting the time identifications into the sampled data group.
Optionally, the predicting, by using a machine learning algorithm, a data change rate of different data types at different time periods includes:
setting a nonlinear regression network, and setting an input layer, a calculation layer and an output layer in the network;
using historical acquisition data as input parameters of an input layer;
setting a training model and an activation function in the calculation layer, wherein the training model comprises a plurality of neural network units, and each neural network unit is provided with a weight;
training the historical collected data serving as sample data in the training model, and acquiring a calculation result;
training the calculation result through the activation function to obtain a final fitting result, and obtaining the data change rates of different data types in different time periods through the final fitting result.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below.
FIG. 1 is a diagram of a multi-source heterogeneous network architecture topology for a smart city;
FIG. 2 is a flow diagram illustrating multi-source heterogeneous data compression, according to an embodiment.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
The smart city can be divided into four levels, the first level is a perception layer, namely data are collected through various city nerve endings (Internet of things equipment), such as various cameras, temperature sensors, humidity sensors, water pressure sensors, mobile terminals and the like, are responsible for acquiring different types of data at different times and different places, the second layer is a communication layer and is responsible for uploading and summarizing the data acquired at different times and different places according to a certain communication protocol, the third layer is a platform layer, after receiving the data, the fourth layer is an application layer, and after useful data is extracted, that is, the services are provided for the aspects of the city, such as intelligent traffic, intelligent buildings, intelligent medical treatment, and intelligent power.
The core of the smart city lies in breaking an information island, so that massive data can be connected, stored and inquired, how to construct a large enough cloud storage, and how to provide a knowledge graph for a decision maker in a short time is a very key core problem.
The data types of the smart city are various, and for convenience of understanding, the most typical three types of data are selected in the embodiment of the invention, namely the geographic information display system GIS, the building information model BIM and the Internet of things IoT. The GIS is an information system that collects, stores, edits, manages, analyzes, shares, and displays geographic data related to the whole or part of the space of the earth's surface layer (including the atmosphere) with the support of a computer hardware and software system. In the city information model CIM, the GIS needs to provide six aspects of capability: 1) establishing a unified coordinate system to align various city information; 2) providing a two-dimensional and three-dimensional integrated base map; 3) managing a link network between BIM monomers, such as roads, pipe corridors, pipelines and the like; 4) providing a topological relation space analysis capability; 5) providing BIM data management capability for large-scale building groups; 6) and the support of various terminals is provided, and the CIM application is met in all directions. BIM is a digital representation of the physical and functional characteristics of a facility that can be used as a shared knowledge resource for facility information, becoming a reliable basis for facility life-cycle decisions. The IOT is an expanded application and network extension of a communication network and the Internet, realizes information interaction and seamless link between people and objects and between objects on the basis of sufficient information interaction and link by carrying out perception recognition, calculation, processing and knowledge mining on the physical world, thereby achieving real-time control, accurate management and scientific decision on the physical world. The network architecture of the Internet of things comprises a sensing layer, a network layer and an application layer. The sensing layer realizes intelligent sensing identification, information acquisition processing and automatic control of the physical world, and connects the physical entity to the network layer and the application layer through the communication module. The network layer mainly realizes the transmission, routing and control of information, and can rely on public telecommunication networks and the Internet and also can rely on industry special communication networks. The application layer comprises application infrastructure middleware and various Internet of things applications, and the application infrastructure middleware provides general basic service facilities, capabilities and resource calling interfaces for the Internet of things applications, such as information processing, calculation and the like, so that various applications of the Internet of things in various fields are realized on the basis.
GIS is generally a spatial model built for urban or regional objects, with the primary purpose of describing urban or regional scale geospatial objects with less focus on detail. BIM focuses on the internal details of a facility or building scale object but is rarely used for objects other than facilities or buildings. Thus, the internal details of the facility or building object are defined and expressed by the BIM model, and the larger size object outside the facility or building is described by the GIS, i.e. the information and functions of the GIS and BIM need to be integrated when describing all physical entities within the area or city size.
In the construction, operation and maintenance stages of engineering, the states of objects such as buildings, equipment, facilities and the like can change dynamically, and the BIM cannot completely meet the requirements on state information acquisition and transmission of the objects. Therefore, the IOT technology needs to be introduced to complete the collection of the dynamic information of the environment and the object and the association of the dynamic information with the static information described by the BIM. The integration of BIM and IOT technologies can link the reality of the virtual and construction operation and maintenance stages in the design stage, so that the consistency and the interchangeability of BIM delivery results in each engineering stage are guaranteed.
The GIS is used as a uniform space carrier of various objects of the Internet of things, and the quantity and quality of information sources directly influence the application range and depth of the Internet of things. During the construction period of the Internet of things, analysis and selection of sensor layout can be realized based on basic geographic information, and the scientificity and rationality of terminal layout are realized. After the sensor network is built, the positioning, tracking, searching and controlling of the sensor can be realized through the basic geographic information platform, and finally all the things-internet objects fall on the uniform space platform, and the required information can be found visually, vividly and vividly on the platform. That is, the GIS is a core technology that can make the internet of things more intelligent, orderly, intuitive and useful.
In the large-scale and large-range application field, by receiving a certain event in a certain place and the change process of the event along with the time, the geographic information technology can judge the change process of the phenomenon according to the existing sensing data, and can backtrack the past and predict the future. Namely, the Internet of things provides a new information acquisition method for the GIS, and the management scope of the GIS is expanded.
Fig. 1 illustrates a multi-source heterogeneous network architecture according to an embodiment of the present invention, as shown in fig. 1, in an embodiment of the present invention, as shown in fig. 1, the multi-source heterogeneous network architecture may be divided into four layers, where a first layer is a sensing layer, and the sensing layer includes various IoT devices, which are used to collect different types of data at different times and different addresses, for example, GIS data, BIM data, and various types of IoT data (for example, temperature, humidity, and image data collected by a temperature and humidity sensor, a video collection terminal, and the like). In the awareness layer, each IOT device is in a multi-Mode Heterogeneous Wireless Network (MHWN) Network. In the embodiment of the invention, the MHWN network is a multi-mode heterogeneous network. The multi-mode heterogeneous network is a network comprising a plurality of types of nodes and a plurality of types of relations, different types of networks which are overlapped with each other are fused together, so that the service diversity requirement of a future terminal is met, and the MHWN network is configured to be capable of dynamically selecting and switching among a plurality of communication networks according to the dynamic and differentiated requirements of communication or services. The second layer is an aggregation layer, namely an access layer, the aggregation layer comprises a plurality of Data Transmission Systems (DTSs), each DTS is responsible for summarizing data reported by all IoT devices in a current cell and transmitting the data to the core layer, the aggregation layer can use a radio technology of a software defined SDN to merge a plurality of communication networks, and a data transmission unit can realize SDR function through one or more communication in the plurality of communication networks, so that a software module can run in an MHWN network. The third layer is a core layer, the core layer is a virtual networking structure of a network slice, that is, a network function virtualization NFV is configured to perform a hierarchical decision on reported data, wherein the NFV is a bottom layer platform architecture of the core layer. An access network of the MHWN network. The fourth layer is an application layer, and after the core layer is analyzed and decided, various applications are performed.
The plurality of communication networks include a broadband network and a narrowband network, the broadband network includes a public mobile communication network including but not limited to at least one of 3G, 4G and 5G networks and/or a wireless local area network including but not limited to a WiFi network, and the narrowband network includes but not limited to at least one of an NB-IoT (Narrow Band Internet of Things) network, an LTE-M (LTE-Machine to Machine) network, and a Long Range Radio (Long Range Radio) network.
Wherein the data transmission unit may function to connect different networks. The data transmission unit can communicate with the multimode wireless base station through one or more of the multiple communication networks because of supporting multiple networks, and can also communicate with the terminal through the communication mode supported by the terminal, thereby realizing network connection between different types of terminals and different types of base stations.
When the network is applied to the internet of things, the terminal can comprise sensing equipment, and the sensing equipment can comprise a sensor and/or an actuator. From a broader field of application, the terminals include, but are not limited to, oblique photography, vehicles (vehicles), ships (shifts), airplanes (aircrafts), satellites (satellites), smart home products (smart home products), meteorological devices (meteorological devices), environmental protection devices (environmental protection devices), sensors (sensors), fire-fighting devices (fire-lighting devices), cell phones (cell phones), medical instruments (medical instruments), and the like.
(1) The terminal transmits its own data to the data transmission unit through its connection (e.g., Bluetooth, Zigbee, Z-wave, WiFi, Wpan, etc.) with the data transmission unit; the embodiment of the invention does not limit the connection mode of the terminal and the data transmission unit.
(2) The data transmission unit transmits the data of the terminal through an MHWN network, wherein the MHWN network can select to transmit the data through various different communication networks such as LTE, WiFi and IoT under the control of the NFV network unit, the NFV can realize intelligent selection of network backhaul based on various options, and the scheduling is not limited to the coverage or capacity condition of the selectable networks and also comprises scheduling based on upper layer service requests or data service characteristics. For example, data from the sensor acquired by the data transmission unit may be transmitted through the IoT network; the video data from the video monitoring device acquired by the data transmission unit can be transmitted through an LTE or WiFi network and the like. The NFV core network unit and the MHWN network may be connected by wire or wirelessly, wherein the LTE may be replaced by a communication standard of 5G or higher, wherein the IOT may include lora (lorawan), NB-IOT, and the like.
Fig. 2 is a flowchart of one multi-source heterogeneous data compression method according to an embodiment of the present invention, and as shown in fig. 2, the method includes:
s101, a data transmission unit DTS collects data uploaded by a plurality of terminal devices, wherein the data types comprise IoT data, GIS data and BIM data, the data comprise time attributes and region attributes, the terminal devices are networked by adopting a multi-source heterogeneous network, and the DTS is positioned on a data aggregation layer;
for convenience of illustration, IoT data is defined as arrays a1, a2, A3, a4 … An, GIS data is arrays B1, B2, B3, B4 … Bn, BIM data is arrays C1, C2, C3, C4 … Cn. In the array, each data has data attribute parameters, i.e. time and space attributes, and the time attribute can be represented by a time attribute parameter, such as t0-tn-1The spatial attributes can be divided according to administrative districts, administrative streets and the like, the dividing format and standard can be customized, but data among different types need to be unified.
S102, grouping and cleaning the acquired data according to different data types by the DTS to generate a matrix type original data group;
taking IoT, GIS and BIM data as examples, DTS groups according to different data types, and divides the data into 3 groups, and the 3 groups of data form a matrix type original data group together. The following were used:
﹛A1，A2,A3,A4…AnB1,B2,B3,B4…BnC1,C2,C3,C4…Cn﹜
wherein A isn＝a1a2a3…an，Bn＝b1b2b3…bnCn＝c1c2c3…cn，That is, each set of data includes data that differs by time and/or region.
S103, the DTS samples the original data set according to a preset sampling model to obtain a plurality of sample data of the original data set, wherein the sample data amount is lower than that of the matrix type original data set;
the DTS performs sampling according to a preset sampling model, and may perform sampling through a complex sampling model or through a self-adaptive sampling model, which will be described below.
The first scheme is as follows: complex sampling model
The starting point of the complex sampling model is as follows: different data types have different data change modes, for example, in IoT data, temperature data collected by a temperature sensor has a large difference between morning and evening, while for GIS data, the data difference is not large in a short period of time, for BIM data, the energy consumption data has a large change according to seasons, and the data change is also large according to different working hours.
The data types can be divided into high-frequency data (data with large data change and fast) and low-frequency data (data with small data change and slow), so that in the data acquired in the same time period, the high-frequency data has higher reference significance due to fast change and large data change, and higher sampling rate needs to be set; and the low-frequency data has low variation and small variation amplitude, so that the requirement can be met only by acquiring once or a plurality of times in a time period. In addition, the data acquisition rate can be set from two dimensions of time, area, and the like, for example, in a rural area, the data acquisition rate is only required to be acquired once in one period, and in a urban area, the data acquisition rate is required to be acquired 3 times or more in one period.
In the current smart city perception layer, due to cost reasons, most of the adopted sensors are standard components, the data acquisition and reporting frequencies are the same, and the generated original data are also the same. However, for decision makers in smart cities, data of different types, time and regions need not be collected and recorded all the time, but only partial data need to be stored according to local conditions and local conditions. Therefore, unlike the independent setting of the sampling frequency in the sensing layer, the embodiment of the present invention proposes a "sampling" (the number of times of extracting samples in the original data set per unit time) manner in the data aggregation layer, which aims to selectively store data by setting the sampling rate for the original data sets collected in the same time period.
Specifically, the DTS sorts the original data sets according to time sequence, and under the condition of the same region and the same data type, based on the corresponding relation between different time sequences and data sizes, obtains the historical change rate of the data in different time periods, sets a first sampling rate of the data which dynamically changes along with time based on the historical change rate, wherein the first sampling rate is in direct proportion to the data change rate,
and the number of the first and second groups,
the DTS sets a second sampling rate of the data based on the region attributes, the second sampling rate corresponding in size to the different region attributes,
and the number of the first and second groups,
the DTS sets a dynamically varying third sampling rate based on the different data types, the third sampling rate corresponding to the data types one-to-one,
the DTS samples the original data set in sequence based on the first sampling rate, the second sampling rate and the third sampling rate, and acquires a plurality of sampled data sets.
For example, the following data sets:
﹛A1，A2,A3,A4…AnB1,B2,B3,B4…BnC1,C2,C3,C4…Cn﹜
three sampling rates may be provided, the first sampling rate being positively correlated to a rate of change of the historical data, the rate of change of the historical data being an average of one or more of the plurality of rates of change over a period of time. The temperature and humidity are typical IoT data, and in a time period (24 hours), the temperature/humidity changes sinusoidally with the time, and the data change rate is high or low, so the first sampling rate is set to be high or low accordingly.
Likewise, the second sampling rate is dynamically changed based on regional attributes, the sampling rates are different in different regions (rural, small-city, and large-city), and the third sampling rate is dynamically changed based on different data types, for example, the data sampling rate of GIS is low and the data sampling rate of IoT is high.
It should be noted that, due to the diversity and complexity of data, three different sampling rates may be used to sample the original data set in sequence, a first sampling rate may be used for time-sensitive data (the data change rate of which changes with time) (which may be data of different types and different regions) to sample, a second sampling rate may be used for space-sensitive data to sample, and a third sampling rate may be used for type-sensitive data to sample.
For example, the temperature, the humidity, the electric energy and the traffic jam degree belong to different regions and different types, but all belong to time-sensitive data (the temperature, the humidity, the electric energy consumption and the traffic jam change obviously along with the change of the peak time of going to and going from work, the change of the peak time, the change of the morning and evening), then, sampling is carried out by adopting a first sampling rate, then, different sampling rates can be set according to different regions (rural areas, cities, towns, small cities, large cities and the like), secondary sampling is carried out by adopting a second sampling rate on the basis of the first sampling, and similarly, according to different data types, third sampling can be carried out by utilizing a third sampling rate, and finally, a group of sampled sample data groups is output.
Scheme II: adaptive sampling
In one embodiment, the core of the adaptive sampling is the data type and reporting frequency thereof, and the sampling rate can be adaptively adjusted according to different data types and reporting frequencies. For example, the GIS data may be heartbeat packets reported once in several hours, and the fire alarm sensor (BIM data) must report heartbeat packets at least twice in one hour, specifically, the DTS sets an adaptive sampling rate based on the different data types and the frequency of heartbeat packets reported by the data, and samples the original data set by the adaptive sampling rate.
In another embodiment, adaptive sampling may also be based on artificial intelligence techniques for prediction and interpolation operations, e.g.,
predicting the data change rate of different data types at different time periods by adopting a machine learning algorithm;
setting a self-adaptive sampling rate based on the predicted data change rate, wherein the self-adaptive sampling rate is in one-to-one correspondence with the data change rate;
sampling the original data set by self-adaptive sampling rate, generating different time marks, and inserting the time marks into the sampled data set.
The machine learning algorithm predicts the data change rate, and can be carried out by the following method:
setting a nonlinear regression network, and setting an input layer, a calculation layer and an output layer in the network;
using historical acquisition data as input parameters of an input layer; wherein the input layer comprises a plurality of historical data change rates, each represented by x;
setting a training model and an activation function in a calculation layer, wherein the training model comprises a plurality of neural network units, and each neural network unit is provided with a weight;
training historical collected data serving as sample data in a training model, and acquiring a calculation result; the neural network unit is called as neuron for short, and the input and output connection between the neurons constructs a relation network, namely the neural network. The calculation of each neural network may be expressed in terms of y ═ w × f (x) + a, where y denotes the calculation result, w denotes the weight, x is the input parameter, a is a fixed constant, and f (x) represents the training function. For a plurality of historical data change rates, x1, x2.. xn can be set, and then the calculation result is y ═ w1*f(x1)+w2*f(x2)+…+wn*f(xn)+n*a。
Training the calculation result through an activation function to obtain a final fitting result, and obtaining the data change rates of different data types in different time periods through the final fitting result. In the field of machine learning, the activation function may adopt two types, ReLu and Sigmoid, and preferably, the ReLu activation function is selected.
And S104, the DTS performs data splicing on the plurality of sample data sets according to a preset format to form a compressed data set, and sends the compressed data set to a core layer.
The embodiment of the present invention may further perform data volume compression again in a short-to-long manner, for example, after data assembly is performed on a plurality of sample data sets according to a preset format, and before a compressed data set is formed, binary conversion is performed on the assembled data set; in the binary data bits, the repeated data are merged into the same data bit, and a sub-digit number is set for representing the number of repetitions of the data bit.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (6)
1. A multi-source heterogeneous data compression method applied to a smart city is characterized by comprising the following steps:
the data transmission method comprises the steps that a data transmission unit DTS acquires data uploaded by a plurality of terminal devices, wherein the data types comprise IoT data, GIS data and BIM data, the data comprise time attributes and region attributes, the terminal devices are networked by adopting a multi-source heterogeneous network, and the DTS is located on a data aggregation layer;
the DTS carries out grouping and data cleaning on the acquired data according to different data types to generate a matrix type original data group;
the DTS samples the original data group according to a preset sampling model to obtain a plurality of sample data of the original data group, wherein the sample data amount is lower than the data amount of the matrix type original data group;
and the DTS performs data splicing on the plurality of sample data sets according to a preset format to form a compressed data set, and sends the compressed data set to the core layer.
2. The method of claim 1, wherein the DTS performs sampling on the original data set according to a preset sampling model, and obtaining a plurality of sample data of the original data set comprises:
the DTS sorts the original data groups according to time sequence, acquires historical change rates of data in different time periods based on corresponding relations between different time sequences and data sizes under the condition of the same region and the same data type, sets a first sampling rate of the data which dynamically changes along with time based on the historical change rates, wherein the first sampling rate is in direct proportion to the data change rate,
and the number of the first and second groups,
the DTS sets a second sampling rate of data based on the region attributes, the second sampling rate having a one-to-one correspondence with the different region attributes,
and the number of the first and second groups,
the DTS sets a dynamically varying third sampling rate based on the different data types, the third sampling rate corresponding one-to-one to the data types,
and the DTS samples the original data group in sequence based on the first sampling rate, the second sampling rate and the third sampling rate, and acquires a plurality of sampled data groups.
3. The method of claim 1, wherein the DTS performs the sampling of the raw data set according to a predetermined sampling model, comprising:
and the DTS sets a self-adaptive sampling rate based on the different data types and the frequency of the data reporting heartbeat packet, and samples the original data group through the self-adaptive sampling rate.
4. The method of claim 1, wherein after said data assembling said plurality of sample data sets according to a predetermined format and before forming a compressed data set, said method further comprises:
carrying out binary conversion on the assembled data group;
and combining repeated data into the same data bit in the binary data bits, and setting a secondary digit for representing the repeated times of the data bits.
5. The method of claim 1, wherein the DTS performs the sampling of the raw data set according to a predetermined sampling model, comprising:
predicting the data change rate of different data types at different time periods by adopting a machine learning algorithm;
setting an adaptive sampling rate based on the predicted data change rate, the adaptive sampling rate corresponding to the data change rate one to one;
and sampling the original data group by the self-adaptive sampling rate, generating different time identifications, and inserting the time identifications into the sampled data group.
6. The method of claim 5, wherein predicting the rate of change of data for different data types at different time periods using a machine learning algorithm comprises:
setting a nonlinear regression network, and setting an input layer, a calculation layer and an output layer in the network;
using historical acquisition data as input parameters of an input layer;
setting a training model and an activation function in the calculation layer, wherein the training model comprises a plurality of neural network units, and each neural network unit is provided with a weight;
training the historical collected data serving as sample data in the training model, and acquiring a calculation result;
training the calculation result through the activation function to obtain a final fitting result, and obtaining the data change rates of different data types in different time periods through the final fitting result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010771901.6A CN111935124A (en) | 2020-08-04 | 2020-08-04 | Multi-source heterogeneous data compression method applied to smart city |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010771901.6A CN111935124A (en) | 2020-08-04 | 2020-08-04 | Multi-source heterogeneous data compression method applied to smart city |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111935124A true CN111935124A (en) | 2020-11-13 |
Family
ID=73306688
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010771901.6A Withdrawn CN111935124A (en) | 2020-08-04 | 2020-08-04 | Multi-source heterogeneous data compression method applied to smart city |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111935124A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112699252A (en) * | 2021-03-25 | 2021-04-23 | 成都数联铭品科技有限公司 | Processing method of attribute data applied to knowledge graph and electronic equipment |
| CN112906956A (en) * | 2021-02-05 | 2021-06-04 | 希盟泰克(重庆)实业发展有限公司 | BIM and CIM combined urban energy consumption prediction method |
| CN114142866A (en) * | 2021-11-26 | 2022-03-04 | 北京人大金仓信息技术股份有限公司 | Data compression method and device, electronic equipment and storage medium |
| CN117076463A (en) * | 2023-10-16 | 2023-11-17 | 环天智慧科技股份有限公司 | Multi-source data aggregation storage system for smart city |
-
2020
- 2020-08-04 CN CN202010771901.6A patent/CN111935124A/en not_active Withdrawn
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112906956A (en) * | 2021-02-05 | 2021-06-04 | 希盟泰克(重庆)实业发展有限公司 | BIM and CIM combined urban energy consumption prediction method |
| CN112699252A (en) * | 2021-03-25 | 2021-04-23 | 成都数联铭品科技有限公司 | Processing method of attribute data applied to knowledge graph and electronic equipment |
| CN112699252B (en) * | 2021-03-25 | 2021-07-23 | 成都数联铭品科技有限公司 | Processing method of attribute data applied to knowledge graph and electronic equipment |
| CN114142866A (en) * | 2021-11-26 | 2022-03-04 | 北京人大金仓信息技术股份有限公司 | Data compression method and device, electronic equipment and storage medium |
| CN117076463A (en) * | 2023-10-16 | 2023-11-17 | 环天智慧科技股份有限公司 | Multi-source data aggregation storage system for smart city |
| CN117076463B (en) * | 2023-10-16 | 2023-12-29 | 环天智慧科技股份有限公司 | Multi-source data aggregation storage system for smart city |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111935124A (en) | Multi-source heterogeneous data compression method applied to smart city | |
| CN111885643A (en) | Multi-source heterogeneous data fusion method applied to smart city | |
| Du et al. | The sensable city: A survey on the deployment and management for smart city monitoring | |
| Wang et al. | Multimedia communications over cognitive radio networks for smart grid applications | |
| CN109284619A (en) | The region Locating Type edge calculations system and method for heterologous city Internet of Things | |
| CN1498382B (en) | System and method for design, tracking, measurement, prediction and optimization of data communications networks | |
| CN105531963B (en) | The system and method that multiple target cell is closed in wireless network | |
| EP3644581B1 (en) | Edge-cloud collaboration system for analyzing internet of things data and operating method thereof | |
| CN106993059A (en) | A kind of agriculture feelings monitoring system based on NB IoT | |
| Bačić et al. | Integrated sensor systems for smart cities | |
| CN106028380A (en) | System for mining, analyzing and displaying wireless network performance parameters and operating method thereof | |
| CN103167414A (en) | Meteorological information service system based on intelligent mobile phone participating in perception and implementing method thereof | |
| CN102780783B (en) | Crop growth environment information real-time perception and dynamically present system and method | |
| Lan et al. | Architecture of wireless sensor networks for environmental monitoring | |
| CN102378410B (en) | Deployment method of wireless sensor nodes of environmental monitoring system in museum internet of things | |
| CN104640081A (en) | Information reminding method, information reminding device and server | |
| Bouroche et al. | Urban computing: The technological framework for smart cities | |
| Chen et al. | Self-aware autonomous city: From sensing to planning | |
| Bouida et al. | Carleton-cisco iot testbed: Architecture, features, and applications | |
| Giacobbe et al. | Arancino. ccTM: An open hardware platform for urban regeneration | |
| Imoize et al. | Development of an energy-efficient wireless sensor network model for perimeter surveillance | |
| Sadio et al. | Rethinking intelligent transportation systems with Internet of Vehicles: Proposition of sensing as a service model | |
| CN108761440A (en) | A kind of ship monitoring method and system based on the radar exploration technique | |
| CN109743351A (en) | A kind of method, apparatus, dustbin and the server of the distribution of adjustment dustbin | |
| KR102448724B1 (en) | Supply resource ai controling system of smart-city based digital twin |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201113 |
|
| WW01 | Invention patent application withdrawn after publication |