CN107590260A - Cloud data real-time search method and its system - Google Patents
Cloud data real-time search method and its system Download PDFInfo
- Publication number
- CN107590260A CN107590260A CN201710860170.0A CN201710860170A CN107590260A CN 107590260 A CN107590260 A CN 107590260A CN 201710860170 A CN201710860170 A CN 201710860170A CN 107590260 A CN107590260 A CN 107590260A
- Authority
- CN
- China
- Prior art keywords
- cloud data
- hbase
- region
- row
- search method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention provides a kind of cloud data real-time search method and its system, it solves the problems such as prior art is low to the recall precision of magnanimity.Comprise the following steps：S1：Map partitioning into some small grids with unique ID character strings, and the ID character strings of more adjacent small grid have more common prefix characters；S2：The cloud data of Hadoop clusters is directed into HBase databases using the ID text string generations rule of small grid in step S1；S3：Parallel search is carried out in Hbase Region Server using the spatial data retrieval method of HBase Endpoint mechanism based on search condition.The present invention has advantages below：The point for making to close on map be stored in HBase databases similar in region, effectively improve recall precision.
Description
Technical field
The present invention relates to geographical spatial data processing and big data treatment technology, more particularly to a kind of cloud data are real-time
Search method and its system.
Background technology
Laser radar technique can with quick obtaining highly dense, high-precision laser point cloud data, pass through post-processing point cloud
Data, the three-dimensional stereo model that precision is Centimeter Level can be established.This technology that becomes more meticulous has been applied to military, civilian each
Level, for example, unmanned, virtual reality, architectural engineering etc..However, because the data volume of this technology collection is huge (generally
For TB levels), how effectively to store and manage these mass datas, the quick processing based on mass data and three-dimensional stereo model
Technical bottleneck also be present, turn into the wide variety of one kind of laser radar technique in modeling, the real-time retrieval based on massive spatial data
Obstacle.
In order to solve the above-mentioned technical problem, people have carried out long-term exploration, such as Chinese patent discloses one kind and is based on
Swing the variable field-of-view three-dimensional reconstruction apparatus [application number of laser radar：CN201610444260.7], including laser radar swing
Mechanism, mechanism kinematic control module and three-dimensional point cloud rebuild module, wherein：Single line laser radar is that three dimensional point cloud collection is set
Standby, laser radar center fixed mechanism bears laser radar weight and fixed laser radar center, laser radar weave control machine
Structure realizes the regulation of laser radar field range and drives its omnidirectional to swing, and three forms laser radar swing mechanism；Mechanism transports
The motion of dynamic control module control laser radar swing mechanism, and laser radar pose is measured in real time；Three-dimensional point cloud weight
Modeling block splices to multiframe lidar measurement data, rebuilds space three-dimensional point cloud.
Such scheme can by the measurement and reconstruction to three-dimensional environment, field range and point cloud are distributed according to demand into
Row regulation, to realize to the focus measurement of diverse location spatial information, have that precision is high, highly reliable, the good feature of adaptability.But
It is there are still part deficiency, for example, based on the real-time retrieval of massive spatial data also in the technical bottleneck stage.
It is especially suitable for storing in the widely used big data framework of internet industry, Hadoop in addition, Hadoop is one kind
With processing mass data, there is high extension, scalability, HBase databases provide the real-time retrieval energy to Hadoop files
Power, existing technical problem during the foregoing real-time retrieval to mass data is can solve the problem that by Hadoop.But although
Hadoop in internet industry extensive use, but Hadoop technologies GIS-Geographic Information System industry successful application also not
It is more, particularly it is in laser radar field, the main difficulty of Hadoop processing laser radar datas：1.Hadoop typical cases
For the processing to text message, the processing application to geographic information data is also not carried out Seamless integration-；2. it is based on spatial information
Retrieval be different from text based retrieval, HBase does not support the function of space real-time retrieval.
The content of the invention
Regarding the issue above, the present invention provides a kind of method is simple, the high cloud data of recall precision is real-time
Search method；
It is another object of the present invention to for above-mentioned technical problem, there is provided a kind of based on cloud data real-time search method
Cloud data real-time retrieval system.
To reach above-mentioned purpose, present invention employs following technical proposal：
The cloud data real-time search method of the present invention, comprises the following steps：
S1：Map partitioning into some small grids with unique ID character strings, and the ID characters of more adjacent small grid
String has more common prefix characters；
S2：The cloud data of Hadoop clusters is directed into using the ID text string generations rule of small grid in step S1
In HBase databases；
S3：Based on search condition using the spatial data retrieval method of HBase Endpoint mechanism in Hbase Region
Parallel search is carried out in Server.
By such scheme, the point for making to close on map is stored in region similar in HBase databases, can effectively improve inspection
Rope efficiency.
In above-mentioned cloud data real-time search method, in step sl, using the map grid ID of GEOHASH algorithms
Map partitioning is some small grids by generation method.
In above-mentioned cloud data real-time search method, in step s 2, the point cloud number being directed into HBase databases
Deposited according to according to its geographical location information to corresponding small grid.
In above-mentioned cloud data real-time search method, in step s 2, the point cloud number being directed into HBase databases
The form of database table preserves according to this, and the corresponding small grid of often row of database table, and RowKey is based on the small grid pair
The GEOHASH codes in region are answered, there is the point cloud information row cluster in the row corresponding region per a line, the row in row cluster preserve one
The information of point.
In above-mentioned cloud data real-time search method, in step s3, described HBase Endpoint mechanism
Spatial data retrieval method is that can be directed to the search method of any closed polygon search condition.
In above-mentioned cloud data real-time search method, in step s3, HBase Endpoint mechanism pair is used
The method that HBase Region Server carry out parallel search includes：
S3.1：RowKey is filtered, obtains the candidate row for the condition that meets；
S3.2：The point included in row to meeting condition judges, returns to retrieval result.
In above-mentioned cloud data real-time search method, in step S3.1, obtaining the method for candidate row includes：
S3.1.1：All small grids of region of search covering are generated based on region of search；
S3.1.2：The GEOHASH codes of the small grid obtained in calculation procedure S3.1.1, so as to generate region to be searched
RowKey lists, candidate row is obtained using the RowKey lists.
In above-mentioned cloud data real-time search method, step S3.2 is specifically included：
The point included in each candidate row is judged whether in region of search, and returns to the point in region of search.
It is further comprising the steps of before step S1 in above-mentioned cloud data real-time search method：
Cloud data is read from client, and the cloud data is write into Hadoop distributed file systems.
A kind of cloud data real-time retrieval system based on cloud data real-time search method.
Cloud data real-time search method of the present invention and its system have advantages below compared to prior art：1st, pass through by
Map partitioning is that the cloud data that some small grids close on geographical position is stored in similar disk space, improves retrieval effect
Rate；2nd, parallel search is carried out by the spatial data retrieval method of HBase Endpoint mechanism, further ensures efficient inspection
Rope, while HBase is supported space real-time retrieval function.
Brief description of the drawings
Fig. 1 is the inventive method flow chart；
Fig. 2 is HBase database table structures in the present invention.
Embodiment
Below using the preferred embodiments of the present invention and with reference to accompanying drawing, technical scheme is made further to retouch
State, but the present invention is not limited to these embodiments.
As shown in figure 1, the cloud data real-time search method of the present invention first carries out following steps：
The cloud data that laser radar is collected into is sent to client, then reads cloud data from client, and will
The cloud data write-in Hadoop distributed file systems in, cloud data write-in Hadoop distributed file systems after perform with
Lower step：
S1：If the map grid ID generation methods of GEOHASH algorithms are used by map partitioning to have unique ID character strings
Small grid；
Below to introduce GEOHASH encryption algorithm exemplified by coordinate (39.92324,116.3906), first by latitude
Scope (- 90,90) is divided equally into two sections (- 90,0), (0,90), if target latitude is located at previous section, is encoded to
0, otherwise it is encoded to 1.Because 39.92324 belongs to (0,90), 1 is encoded to so taking, (0,90) is then divided into (0,45) again,
(45,90) two sections, and 39.92324 are located at (0,45), so be encoded to 0, by that analogy, until precision meet the requirements for
Only, obtain latitude and be encoded to 1,011 1,000 1,100 0,111 1001；Longitude is obtained with same method to encode.
The characteristics of this method, is and the ID character strings of more adjacent small grid have more common prefix characters.
S2：The cloud data of Hadoop clusters is directed into using the ID text string generations rule of small grid in step S1
In HBase databases, that is, the cloud data being directed into HBase databases is deposited to corresponding small according to its geographical location information
In grid, and the cloud data being directed into HBase databases is preserved in the form of database table, and HBase database tables are such as
Shown in Fig. 2, the corresponding sub-regions of a line, the corresponding zonule (small grid) of a, often row, RowKey (line unit) is based on this
The GEOHASH codes in region；B, there is the point cloud information row cluster in the row corresponding region per a line, the row in row cluster preserve one
The information of point.Based on this design method, that is closed on map is stored in region similar in database, can improve retrieval effect
Rate.
S3：Based on search condition using the spatial data retrieval method of HBase Endpoint mechanism in Hbase Region
Parallel search is carried out in Server.Also, the spatial data retrieval method of the HBase Endpoint mechanism of the present embodiment is energy
Enough it is directed to the search method of any closed polygon search condition.
Art personnel should be known that the information that HBas data are preserved in Region, and Region Server are
The server 1 that HBase is operated on each working node, for safeguarding Region state, there is provided the management for Region
And service；Hbase has two kinds of secondary processors, and one kind is Observer (observer), similar to the trigger of relational database
(trigger), another is exactly EndPoint here, similar to the storing process of relational database, specifically, this implementation
Being carried out the method for parallel search in example to HBase Region Server using HBase Endpoint mechanism is included：
S3.1：RowKey is filtered, obtains the candidate row for the condition that meets；
Wherein, obtaining the method for candidate row includes：
All small grids of region of search covering are generated based on region of search；Calculate the small grid obtained in abovementioned steps
GEOHASH codes, so as to generate the RowKey lists in region to be searched, use the RowKey lists obtain candidate row；
S3.2：The point included in row to meeting condition judges, returns to retrieval result.
And step S3.2 is specifically included：The point included in each candidate row is judged whether in region of search, and returned
Point in region of search.
Further, the present embodiment also discloses a kind of cloud data based on cloud data real-time search method and examined in real time
Cable system.
Specific embodiment described herein is only to spirit explanation for example of the invention.Technology belonging to the present invention is led
The technical staff in domain can be made various modifications or supplement to described specific embodiment or be replaced using similar mode
Generation, but without departing from the spiritual of the present invention or surmount scope defined in appended claims.
Although more having used the terms such as small grid, line unit, candidate row, cloud data herein, it is not precluded from using
The possibility of other terms.It is used for the purpose of more easily describing and explaining the essence of the present invention using these terms；Them
Any additional limitation is construed to all to disagree with spirit of the present invention.
Claims (10)
1. a kind of cloud data real-time search method, it is characterised in that comprise the following steps：
S1：Map partitioning into some small grids with unique ID character strings, and the ID character strings of more adjacent small grid have
There is more common prefix characters；
S2：The cloud data of Hadoop clusters is directed into HBase using the ID text string generations rule of small grid in step S1
In database；
S3：Based on search condition using the spatial data retrieval method of HBase Endpoint mechanism in Hbase Region
Parallel search is carried out in Server.
2. the cloud data real-time search method according to right will go 1, it is characterised in that in step sl, use
Map partitioning is some small grids by the map grid ID generation methods of GEOHASH algorithms.
3. the cloud data real-time search method according to right will go 2, it is characterised in that in step s 2, be directed into
Cloud data in HBase databases is deposited to corresponding small grid according to its geographical location information.
4. the cloud data real-time search method according to right will go 3, it is characterised in that in step s 2, be directed into
Cloud data in HBase databases is preserved in the form of database table, and the corresponding small grid of often row of database table,
RowKey is the GEOHASH codes based on the small grid corresponding region, has the point cloud information row in the row corresponding region per a line
Cluster, the row in row cluster preserve the information of a point.
5. the cloud data real-time search method according to right will go 4, it is characterised in that in step s3, described
The spatial data retrieval method of HBase Endpoint mechanism is that can be directed to the retrieval side of any closed polygon search condition
Method.
6. the cloud data real-time search method according to right will go 1, it is characterised in that in step s3, use HBase
The method that Endpoint mechanism carries out parallel search to HBase Region Server includes：
S3.1：RowKey is filtered, obtains the candidate row for the condition that meets；
S3.2：The point included in row to meeting condition judges, returns to retrieval result.
7. the cloud data real-time search method according to right will go 6, it is characterised in that in step S3.1, waited
The method of choosing row includes：
S3.1.1：All small grids of region of search covering are generated based on region of search；
S3.1.2：The GEOHASH codes of the small grid obtained in calculation procedure S3.1.1, so as to generate the RowKey in region to be searched
List, candidate row is obtained using the RowKey lists.
8. the cloud data real-time search method according to right will go 7, it is characterised in that step S3.2 is specifically included：
The point included in each candidate row is judged whether in region of search, and returns to the point in region of search.
9. the cloud data real-time search method according to right will go 1, it is characterised in that before step S1 also include with
Lower step：
Cloud data is read from client, and the cloud data is write into Hadoop distributed file systems.
10. a kind of cloud data of the cloud data real-time search method based on described in claim 1-9 any one is examined in real time
Cable system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710860170.0A CN107590260B (en) | 2017-09-21 | 2017-09-21 | Point cloud data real-time retrieval method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710860170.0A CN107590260B (en) | 2017-09-21 | 2017-09-21 | Point cloud data real-time retrieval method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107590260A true CN107590260A (en) | 2018-01-16 |
| CN107590260B CN107590260B (en) | 2020-02-18 |
Family
ID=61046998
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710860170.0A Active CN107590260B (en) | 2017-09-21 | 2017-09-21 | Point cloud data real-time retrieval method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107590260B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109902115A (en) * | 2019-01-28 | 2019-06-18 | 中山大学 | A kind of region towards raster data and basin Data programming extracting method |
| CN109946724A (en) * | 2019-03-29 | 2019-06-28 | 江苏小牛电动科技有限公司 | A kind of GPS static drift modification method and its device based on GEOHASH algorithm |
| CN112365399A (en) * | 2020-10-09 | 2021-02-12 | 北京星闪世图科技有限公司 | Fan blade image panoramic stitching method and system based on deep learning |
| CN112365399B (en) * | 2020-10-09 | 2024-05-03 | 江苏星闪世图科技(集团)有限公司 | Deep learning-based panoramic stitching method and system for fan blade images |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2339481A1 (en) * | 2009-12-03 | 2011-06-29 | National Digital Research Centre | Enablement of three-dimensional hosting, indexing, analysing and querying structure for spatial systems |
| CN103955511A (en) * | 2014-04-30 | 2014-07-30 | 华南理工大学 | Cloud platform data organization and retrieval method for 3D (three-dimensional) urban building data |
| CN104376053A (en) * | 2014-11-04 | 2015-02-25 | 南京信息工程大学 | Storage and retrieval method based on massive meteorological data |
| CN104820714A (en) * | 2015-05-20 | 2015-08-05 | 国家电网公司 | Mass small tile file storage management method based on hadoop |
| CN106156332A (en) * | 2016-07-06 | 2016-11-23 | 福建富士通信息软件有限公司 | The method screening vehicles passing in and out based on section seclected time and selection area |
-
2017
- 2017-09-21 CN CN201710860170.0A patent/CN107590260B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2339481A1 (en) * | 2009-12-03 | 2011-06-29 | National Digital Research Centre | Enablement of three-dimensional hosting, indexing, analysing and querying structure for spatial systems |
| CN103955511A (en) * | 2014-04-30 | 2014-07-30 | 华南理工大学 | Cloud platform data organization and retrieval method for 3D (three-dimensional) urban building data |
| CN104376053A (en) * | 2014-11-04 | 2015-02-25 | 南京信息工程大学 | Storage and retrieval method based on massive meteorological data |
| CN104820714A (en) * | 2015-05-20 | 2015-08-05 | 国家电网公司 | Mass small tile file storage management method based on hadoop |
| CN106156332A (en) * | 2016-07-06 | 2016-11-23 | 福建富士通信息软件有限公司 | The method screening vehicles passing in and out based on section seclected time and selection area |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109902115A (en) * | 2019-01-28 | 2019-06-18 | 中山大学 | A kind of region towards raster data and basin Data programming extracting method |
| CN109902115B (en) * | 2019-01-28 | 2022-03-04 | 中山大学 | Method for programmatically extracting region and watershed data oriented to raster data |
| CN109946724A (en) * | 2019-03-29 | 2019-06-28 | 江苏小牛电动科技有限公司 | A kind of GPS static drift modification method and its device based on GEOHASH algorithm |
| CN109946724B (en) * | 2019-03-29 | 2024-01-16 | 江苏小牛电动科技有限公司 | GPS static drift correction method and device based on GEOHOASH algorithm |
| CN112365399A (en) * | 2020-10-09 | 2021-02-12 | 北京星闪世图科技有限公司 | Fan blade image panoramic stitching method and system based on deep learning |
| CN112365399B (en) * | 2020-10-09 | 2024-05-03 | 江苏星闪世图科技(集团)有限公司 | Deep learning-based panoramic stitching method and system for fan blade images |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107590260B (en) | 2020-02-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107515952A (en) | The method and its system of cloud data storage, parallel computation and real-time retrieval | |
| CN105426491B (en) | A kind of search method and system of space-time geography big data | |
| Lv et al. | Virtual reality smart city based on WebVRGIS | |
| Li et al. | Big data in smart cities | |
| CN104199986B (en) | Vector data space index method based on hbase and geohash | |
| CN104680331A (en) | IFC based three-dimensional digital management platform for transmission and transformation project | |
| CN105808672A (en) | Browser based mass three-dimensional point cloud data release method | |
| CN104167021B (en) | Space target indexing method based on terrestrial space three-dimensional partition | |
| Pan et al. | Real-time collision detection and distance computation on point cloud sensor data | |
| Wang et al. | Research and implementation on spatial data storage and operation based on Hadoop platform | |
| CN103403769A (en) | Planetary scale object rendering | |
| CN103345508A (en) | Data storage method and system suitable for social network graph | |
| CN103955511A (en) | Cloud platform data organization and retrieval method for 3D (three-dimensional) urban building data | |
| CN105160707A (en) | Three-dimensional model fast visualization method based on viewpoint indexes | |
| CN110532444A (en) | A kind of higher-dimension electric power big data method for visualizing and system | |
| Du et al. | Spatio-temporal data index model of moving objects on fixed networks using hbase | |
| Wang et al. | Efficient visibility analysis for massive observers | |
| Wang et al. | A fast candidate viewpoints filtering algorithm for multiple viewshed site planning | |
| CN107590260A (en) | Cloud data real-time search method and its system | |
| Kritter et al. | On the real-world applicability of state-of-the-art algorithms for the optimal camera placement problem | |
| Bonnell et al. | Constructing material interfaces from data sets with volume-fraction information | |
| Brown et al. | Height and facet extraction from LiDAR point cloud for automatic creation of 3D building models | |
| CN107679127A (en) | Point cloud information parallel extraction method and its system based on geographical position | |
| Nguyen et al. | B-EagleV: visualization of big point cloud datasets in civil engineering using a distributed computing solution | |
| CN104866687B (en) | Support the dynamically spatial-data index construction method of STL data source |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |