CN111078956A - Smart pipe network routing inspection track distribution reduction storage query method - Google Patents
Smart pipe network routing inspection track distribution reduction storage query method Download PDFInfo
- Publication number
- CN111078956A CN111078956A CN201911305059.0A CN201911305059A CN111078956A CN 111078956 A CN111078956 A CN 111078956A CN 201911305059 A CN201911305059 A CN 201911305059A CN 111078956 A CN111078956 A CN 111078956A
- Authority
- CN
- China
- Prior art keywords
- data
- lat
- lng
- standard
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a storage and query method for distribution and shrinkage of routing inspection tracks of an intelligent pipe network, which comprises the following steps of: pre-defining data, defining a standard precision constant R and a cluster radius constant E, acquiring original track data, and packaging according to a p (lng, lat, w, R) format standard; preliminarily diluting the standard data, and processing according to the r data value; the cluster congestion is sparse, the data unit calculates circularly according to the lng lat, cluster blocks with the value smaller than the cluster radius E are divided, and the central point and the weight of each cluster block are calculated after the blocks are divided; data conversion, namely converting the data subjected to clustering processing into JSON data, creating an index directory according to the time of the original data to store the JSON data and releasing the JSON data; and (4) data query, namely retrieving the published JSON file directly according to the time directory index. The invention provides a track data thinning, storing and inquiring method, which solves the problems of slow display speed and rendering overload of track analysis large data volume.
Description
Technical Field
The invention relates to the technical field of intelligent pipe network track query, in particular to a method for storing and querying distribution reduction of an intelligent pipe network routing inspection track.
Background
In recent years, domestic pipe network construction is rapidly developed, and the fields of drainage pipe networks, water supply pipe networks, gas pipe networks, communication cable pipe networks and the like are related. Pipe network operation is an important guarantee measure for ensuring the healthy development of the pipe network. In the daily operation of a pipe network, pipe network inspection is one of important works in pipe network operation. At present, the pipe network inspection mainly depends on manual inspection, and inspection auxiliary workers comprise a storage battery car, an operation guarantee car and the like.
At present, the internet and internet of things industries are rapidly developed and applied in various industries, and the platform operation of a pipe network is in an increasing trend. In order to improve the pipe network inspection efficiency, the construction of a smart pipe network platform is enhanced by means of the internet of things technology, inspection personnel and vehicles are tracked on line, inspection work is monitored and guaranteed to be completely covered without omission, and safe and healthy operation of a pipe network is guaranteed. The intelligent pipe network platform can check the real-time positions of the patrol personnel and vehicles, the historical tracks and the patrol distribution range of the whole personnel on line, and check whether the patrol work is omitted.
The conventional intelligent pipe network platform has a large operation range, needs a large amount of polling personnel and vehicles, generates ten million-level track data every day, and causes certain pressure for the platform to store and inquire track distribution data. How to quickly check the patrol working condition of the whole area needs a storage and query method for distribution and shrinkage of a patrol track of an intelligent pipe network, and the working efficiency of a platform is improved.
An effective solution to the problems in the related art has not been proposed yet.
Disclosure of Invention
The invention aims to provide a storage and query method for distribution and shrinkage of a routing inspection track of a smart pipe network, and the method is used for solving the problems that the track data of the smart pipe network is difficult to store and fast query in the prior art.
In order to achieve the purpose, the invention provides the following technical scheme: a smart pipe network routing inspection track distribution reduction storage query method comprises the following steps:
s1: pre-defining data, defining a standard precision constant R and a cluster radius constant E, acquiring track original data and packaging according to a p (lng, lat, w, R) format standard;
s2: denoising standard data, namely discarding data elements with r values higher than the standard precision according to the standard precision range, and merging w according to the similarity of the lng data values and the lat data values;
s3: clustering and thinning, clustering and grouping standard data and set data according to rules, calculating the central point and weight of each grouping, performing thinning conversion on clustering blocks, and converting one clustering block into one data element;
s4: data conversion, namely converting the data subjected to cluster processing into JSON data, and storing and releasing the JSON data according to a time index format;
s5: and data query, namely searching and issuing JSON files for webpage rendering according to the time index.
As a preferred technical solution of the present invention, the data pre-defining process includes the steps of:
s101: defining a standard precision constant R;
s102: defining a cluster radius constant E;
s103: defining a standard data format as p (lng, lat, w, r), wherein lng represents longitude, lat represents latitude, w represents weight, and r represents data accuracy;
s102: all track data is encapsulated in this format, with w default to 1.
As a preferred technical scheme of the invention, the standard data denoising method comprises the following steps:
s201: circulating the standard data set, judging the R value of each standard data element, and deleting the data units with the R value larger than R in the standard data set;
s202: comparing the standard data one by one, judging the lng and the lat, and adding the w values of the two data units when the lng and the lat are the same to generate a new data unit.
As a preferred technical solution of the present invention, the cluster rarefying comprises the following steps:
s301: creating a first cluster block, randomly taking a data unit from a standard data set, creating a first cluster block set, and deleting the data unit in the standard data set;
s302: clustering, namely taking out one data element pi (long, lat, r and w) from a standard data set one by one, performing calculation of distance with any element in each cluster block, when the distance of any element in a single cluster block is smaller than or equal to E, putting the pi (long, lat, r and w) into the current cluster block, finishing the current calculation, and calculating the next data element; if the conditions are not met with all the cluster block calculation, a new cluster block is created on the basis of the data element;
s303: calculating the center Point of the cluster block, and calculating the center Point (lng, lat) according to all unit data in the cluster block; s is the data volume of the cluster block unit;
s304: and (3) performing thinning conversion on the cluster blocks, wherein one cluster block is converted into one data element p (ng, lat, w), wherein the ng is the lng value of the Point center (ng, lat), the lat is the lat value of the Point center (ng, lat), and the w value is the data amount of the cluster block unit.
As a preferred technical solution of the present invention, the data conversion includes the steps of:
s401: p (lng, lat, w) object data conversion, JSON format sample is:
{{"lng":lng,"lat":lat,"w":w}，{"lng":lng,"lat":lat,"w":w}}；
s402, creating a time file index directory in a format of YY/MM/DD/creating the directory;
s403, JSON data output storage, JSON data printing to a corresponding index directory, file naming: filename json.
As a preferred technical scheme of the invention, the data query process comprises the following steps:
s501: and (4) data query, namely directly returning JSON data for front-end use according to the query date input by the user.
Compared with the prior art, the invention has the following beneficial effects:
(1) the method for inquiring the distribution shrinkage of the routing inspection track of the intelligent pipe network is provided, and the problems of large storage capacity of the distribution data of the inquiry track and low inquiry speed are solved;
(2) the data denoising and data clustering technology is adopted and converted into JSON storage, so that the storage and front-end use analysis time is reduced, and the application use efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a flowchart of a method for searching for a reduced distribution and storage of routing inspection trajectory of a smart pipe network according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating conversion of standardized data set thinning into cluster blocks of the smart pipe network routing inspection track distribution reduction storage query method according to the embodiment of the invention.
Detailed Description
The invention is further described with reference to the following drawings and detailed description:
the first embodiment is as follows:
referring to fig. 1, a method for storing and querying distribution and shrinkage of routing inspection tracks of a smart pipe network according to an embodiment of the present invention includes the following steps:
s1: pre-defining data, defining a standard precision constant R and a cluster radius constant E, acquiring track original data and packaging according to a p (lng, lat, w, R) format standard; the data pre-defining process comprises the following steps:
s101: defining a standard precision constant R, wherein the positioning precision is different due to different strengths of environmental signals where the field track positioning equipment is located, a precision range can be defined according to requirements and used for filtering original data, and the range (20,200) can be defined by the R;
s102: defining a cluster radius constant E, a cluster semi-solution rarefaction strength, and a range (60,100) defined by E;
s103: defining a standard data format as p (lng, lat, w, r), wherein lng represents longitude, lat represents latitude, w represents weight, and r represents data accuracy;
s104: and acquiring track raw data of a certain day on a specified date, packaging the data according to a p (lng, lat, w, r) format standard, and creating a standardized data set.
S2: denoising standard data, namely discarding data elements with r values higher than standard precision according to a standard precision range, and merging w according to similarity of lng and lat data values, wherein the standard data denoising comprises the following steps:
s201: circularly taking out data elements pn (lng, lat, w, R) from the standardized data set, judging the value of each standard data element R, and deleting data units with the R value larger than R in the standard data set;
s202: comparing the standard data one by one, judging the lng and the lat, and adding the w values of the two data units when the lng and the lat are the same to generate a new data unit.
S3: clustering thinning, as shown in fig. 2, standard data and aggregate data are clustered according to rules, the central point and weight of each group are calculated, clustering blocks are thinned and converted, and one clustering block is converted into one data element; the cluster congestion thinning comprises the following steps:
s301: creating a cluster block set C;
s302: creating a first cluster block C0, taking any element from the standard data set, adding the element into C0, and deleting the element from the standard data set;
s303: and sequentially taking out data elements, namely pn (long, lat, w, r), from the standard data set, sequentially performing distance calculation with the data elements pi (long, lat, w, r) of each cluster block of the cluster block set C to obtain a distance, comparing the distance with a constant E, and when the distance calculation between pn (long, lat, w, r) and all data elements in one cluster block is smaller than the constant E, adding pi (long, lat, w, r) to the cluster block, and ending the calculation. If the calculated cluster blocks and all the calculated cluster blocks meet the conditions, a new cluster block is created, and a first element pn (lng, lat, w, r) is added;
s304: calculating the circle center of each cluster block in the cluster block set, and calculating according to all data elements lng lat values of the cluster blocks; the calculation details are as follows:
X＝(Math.Cos(lat0*Math.PI/180)*Math.Cos(lng0*Math.PI/180)+Math.Cos(lat1*Math.PI/180)*Math.Cos(lng1*Math.PI/180)+……+Math.Cos(lats*Math.PI/180)*Math.Cos(lngs**Math.PI/180))/S
Y＝(Math.Cos(lat0*Math.PI/180)*Math.Sin(lng0*Math.PI/180)+Math.Cos(lat1*Math.PI/180)*Math.Sin(lng1*Math.PI/180)+……+Math.Cos(lats*Math.PI/180)*Math.Sin(lngs**Math.PI/180))/S
Z＝(Math.Sin(lat0)+Math.Sin(lat1)+……+Math.Sin(lats))/S
circle center longitude lng ═ Math. Atan2(Y, X)
Circle center latitude Lat ═ Math.Atan2(Z, Math.Sqrt (X + Y))
The weight w value of the cluster blocks is the number of the elements of the cluster blocks, the circle center representation method POINT (lng, lat, w) is adopted, the circle center calculation result of each cluster block is formed into a new data set POINT, and the thinning process is finished.
S4: data conversion, converting the data after the clustering processing into JSON data, storing and releasing the JSON data according to a time index format, wherein the data conversion comprises the following steps:
s401: JOSN conversion, which converts POINT into JSON data, and the format is as follows:
{{"lng":lng,"lat":lat,"w":w}，{"lng":lng,"lat":lat,"w":w}}；
s402: creating a catalog index, creating a time index catalog under a WEB service root catalog according to the date of obtaining original track data, and creating the catalog according to a format YY/MM/DD;
s403: and storing JSON data, namely writing the JSON data into a file under a specified index directory, and naming the file by using YYMMDD.
Step five: and data query, namely searching and issuing a JSON file for webpage rendering according to the time index, wherein the data query process comprises the following steps:
s501: and acquiring the query time input by the user, detecting whether the file exists or not by the front end according to the input date, and directly acquiring JSON data online if the file exists.
The invention provides a storage and query method for routing inspection track distribution shrinkage of a smart pipe network, which solves the problems of large storage capacity of query track distribution data and low query speed; the data denoising and data clustering technology is adopted and converted into JSON storage, so that the storage and front-end use analysis time is reduced, and the application use efficiency is improved.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (6)
1. A smart pipe network routing inspection track distribution reduction storage query method is characterized by comprising the following steps:
s1: pre-defining data, defining a standard precision constant R and a cluster radius constant E, acquiring track original data and packaging according to a p (lng, lat, w, R) format standard;
s2: denoising standard data, namely discarding data elements with r values higher than the standard precision according to the standard precision range, and merging w according to the similarity of the lng data values and the lat data values;
s3: clustering and thinning, clustering and grouping standard data and set data according to rules, calculating the central point and weight of each grouping, performing thinning conversion on clustering blocks, and converting one clustering block into one data element;
s4: data conversion, namely converting the data subjected to cluster processing into JSON data, and storing and releasing the JSON data according to a time index format;
s5: and data query, namely searching and issuing JSON files for webpage rendering according to the time index.
2. The intelligent pipe network routing inspection trajectory distribution reduction storage query method according to claim 1, wherein the data pre-defining process comprises the following steps:
s101: defining a standard precision constant R;
s102: defining a cluster radius constant E;
s103: defining a standard data format as p (lng, lat, w, r), wherein lng represents longitude, lat represents latitude, w represents weight, and r represents data accuracy;
s104: all track data is encapsulated in this format, with w default to 1.
3. The intelligent pipe network routing inspection trajectory distribution reduction storage query method according to claim 1, wherein the standard data denoising comprises the following steps:
s201: circulating the standard data set, judging the R value of each standard data element, and deleting the data units with the R value larger than R in the standard data set;
s202: comparing the standard data one by one, judging the lng and the lat, and adding the w values of the two data units when the lng and the lat are the same to generate a new data unit.
4. The intelligent pipe network routing inspection track distribution reduction storage query method according to claim 1, wherein the clustering rarefaction comprises the following steps:
s301: creating a first cluster block, randomly taking a data unit from a standard data set, creating a first cluster block set, and deleting the data unit in the standard data set;
s302: clustering, namely taking out one data element pi (long, lat, r and w) from a standard data set one by one, performing calculation of distance with any element in each cluster block, when the distance of any element in a single cluster block is smaller than or equal to E, putting the pi (long, lat, r and w) into the current cluster block, finishing the current calculation, and calculating the next data element; if the conditions are not met with all the cluster block calculation, a new cluster block is created on the basis of the data element;
s303: calculating the center Point of the cluster block, and calculating the center Point (lng, lat) according to all unit data in the cluster block; s is the data volume of the cluster block unit;
s304: and (3) performing thinning conversion on the cluster blocks, wherein one cluster block is converted into one data element p (ng, lat, w), wherein the ng is the lng value of the Point center (ng, lat), the lat is the lat value of the Point center (ng, lat), and the w value is the data amount of the cluster block unit.
5. The intelligent pipe network routing inspection trajectory distribution reduction storage query method according to claim 1, wherein the data conversion comprises the following steps:
s401: p (lng, lat, w) object data conversion, JSON format sample is:
{{"lng":lng,"lat":lat,"w":w}，{"lng":lng,"lat":lat,"w":w}}；
s402, creating a time file index directory in a format of YY/MM/DD/creating the directory;
s403, JSON data output storage, JSON data printing to a corresponding index directory, file naming: filename json.
6. The intelligent pipe network routing inspection track distribution reduction storage query method according to claim 1, wherein the data query process comprises the following steps:
s501: and (4) data query, namely directly returning JSON data for front-end use according to the query date input by the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911305059.0A CN111078956B (en) | 2019-12-17 | 2019-12-17 | Smart pipe network routing inspection track distribution reduction storage query method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911305059.0A CN111078956B (en) | 2019-12-17 | 2019-12-17 | Smart pipe network routing inspection track distribution reduction storage query method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111078956A true CN111078956A (en) | 2020-04-28 |
| CN111078956B CN111078956B (en) | 2022-08-05 |
Family
ID=70315369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911305059.0A Active CN111078956B (en) | 2019-12-17 | 2019-12-17 | Smart pipe network routing inspection track distribution reduction storage query method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078956B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111722252A (en) * | 2020-05-14 | 2020-09-29 | 江苏北斗卫星应用产业研究院有限公司 | High-precision operation surface identification method based on track data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2278498A2 (en) * | 2009-07-22 | 2011-01-26 | Xerox Corporation | Scalable indexing for layout based document retrieval and ranking |
| CN106897374A (en) * | 2017-01-19 | 2017-06-27 | 浙江大学 | A kind of personalized recommendation method based on track big data K-NN search |
| CN107169103A (en) * | 2017-05-17 | 2017-09-15 | 南京焦点领动云计算技术有限公司 | A kind of universal architecture data store query method and system |
| CN107766808A (en) * | 2017-09-30 | 2018-03-06 | 北京泓达九通科技发展有限公司 | The method and system that Vehicle Object motion track clusters in road network space |
| US20190325198A1 (en) * | 2015-09-22 | 2019-10-24 | ImageSleuth, Inc. | Surveillance and monitoring system that employs automated methods and subsystems that identify and characterize face tracks in video |
-
2019
- 2019-12-17 CN CN201911305059.0A patent/CN111078956B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2278498A2 (en) * | 2009-07-22 | 2011-01-26 | Xerox Corporation | Scalable indexing for layout based document retrieval and ranking |
| US20190325198A1 (en) * | 2015-09-22 | 2019-10-24 | ImageSleuth, Inc. | Surveillance and monitoring system that employs automated methods and subsystems that identify and characterize face tracks in video |
| CN106897374A (en) * | 2017-01-19 | 2017-06-27 | 浙江大学 | A kind of personalized recommendation method based on track big data K-NN search |
| CN107169103A (en) * | 2017-05-17 | 2017-09-15 | 南京焦点领动云计算技术有限公司 | A kind of universal architecture data store query method and system |
| CN107766808A (en) * | 2017-09-30 | 2018-03-06 | 北京泓达九通科技发展有限公司 | The method and system that Vehicle Object motion track clusters in road network space |
Non-Patent Citations (2)
| Title |
|---|
| YINFU FENG等: ""Mining Spatial-Temporal Patterns and Structural Sparsity for Human Motion Data Denoising"", 《IEEE TRANSACTIONS ON CYBERNETICS ( VOLUME: 45, ISSUE: 12, DEC. 2015)》 * |
| 左志奇等: ""立体匹配点云数据融合去噪算法"", 《电子技术与软件工程》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111722252A (en) * | 2020-05-14 | 2020-09-29 | 江苏北斗卫星应用产业研究院有限公司 | High-precision operation surface identification method based on track data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111078956B (en) | 2022-08-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109189949B (en) | A kind of population distribution calculation method | |
| CN107506393A (en) | A kind of agriculture big data model and its in application agriculturally | |
| CN109670843A (en) | Data processing method, device, computer equipment and the storage medium of complaint business | |
| CN108009937A (en) | A kind of appraisal procedure of distribution main equipment health status | |
| CN105045863B (en) | A kind of method and system for Entities Matching | |
| CN111078956B (en) | Smart pipe network routing inspection track distribution reduction storage query method | |
| CN105574541A (en) | Compactness sorting based network community discovery method | |
| CN111008870A (en) | Regional logistics demand prediction method based on PCA-BP neural network model | |
| CN103674048A (en) | Dynamic intelligent navigation method based on 3G (3-Generation) network | |
| CN109034286A (en) | Based on BIM and RFID construction risk source identification and management method | |
| CN105574265B (en) | Entire assembly model quantitative description towards model index | |
| CN111190982A (en) | Shared bicycle management method and system based on big data | |
| CN109598056A (en) | Measurement Method, system and the storage medium of town site form compactness | |
| CN111002975B (en) | Vehicle energy management method, system, electronic device, and storage medium | |
| CN103177189A (en) | Public source position check-in data quality analysis method | |
| CN113409578B (en) | Traffic network health portrait method and system based on fuzzy clustering | |
| CN106980029B (en) | Vehicle overspeed judgment method and system | |
| CN112948769B (en) | City circle range determining method and system based on commuting big data | |
| CN115909750A (en) | Accident multi-occurrence identification method, device, equipment and storage medium | |
| CN115358504A (en) | National logistics city pivotal sorting model under complex network view angle | |
| CN112633592A (en) | Vehicle frequent-running route calculation method and system based on machine learning clustering algorithm | |
| CN111564035A (en) | Dynamic road condition fusion expression method, device and storage medium | |
| CN104063398A (en) | Recommendation and query method for public transport transfer line | |
| CN110619134A (en) | Integrated detection and restoration method for solving problem of flying spot and spot density of road network data | |
| CN110163492B (en) | Index processing method and device of rail transit energy management system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20200428 Assignee: Anhui Xingtai Financial Leasing Co.,Ltd. Assignor: USTC-GZ INFORMATION TECHNOLOGY CO.,LTD. Contract record no.: X2023980035562 Denomination of invention: A method for reducing storage and querying the distribution of inspection trajectories in smart pipeline networks Granted publication date: 20220805 License type: Exclusive License Record date: 20230518 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A method for reducing storage and querying the distribution of inspection trajectories in smart pipeline networks Effective date of registration: 20230519 Granted publication date: 20220805 Pledgee: Anhui Xingtai Financial Leasing Co.,Ltd. Pledgor: USTC-GZ INFORMATION TECHNOLOGY CO.,LTD. Registration number: Y2023980041171 |