CN106599241A - Big data visual management method for GIS software - Google Patents
Big data visual management method for GIS software Download PDFInfo
- Publication number
- CN106599241A CN106599241A CN201611182291.6A CN201611182291A CN106599241A CN 106599241 A CN106599241 A CN 106599241A CN 201611182291 A CN201611182291 A CN 201611182291A CN 106599241 A CN106599241 A CN 106599241A
- Authority
- CN
- China
- Prior art keywords
- data
- big data
- visual
- big
- gis software
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
Abstract
The invention discloses a big data visual management method for GIS software. The method comprises the following steps: 1) constructing distributed data sources applicable for different data storage modes; 2) according to each data storage mode, inputting known parameters to open the corresponding data source, and accessing and reading big data stored in a server; 3) achieving visual data management operations for the read big data; and 4) uploading the processed data to the server for data warehousing or sharing with others. The method provided by the invention has the advantages that data clusters can be conveniently and visually operated and managed through the interactive operations, so that data analysis effects can be directly displayed, and better data understanding of common users, deeper analysis of data analysis experts and decision-making of managers can be facilitated.
Description
Technical field
The present invention relates to field of computer technology and GIS-Geographic Information System field, and in particular to be directed in a kind of GIS software
The visual management method of big data.
Background technology
With the arriving of cloud era, big data（BigData）Also increasing concern has been attracted.And big data and its skill
Value content, excavating cost in art is more even more important than quantity.For numerous industries, how to be using these large-scale datas
It is crucial.The storage and process of big data is particularly important.Big data process, can will be carried out with being worth to be oriented to big data
The various process such as processing, excavation and optimization.
Big data is that a kind of scale is arrived greatly at aspects such as acquisition, storage, management, analyses well beyond traditional database software
The data acquisition system of means capability scope, it has big data scale, quick stream compression, various data type and value
The low four big feature of density.The core of big data technology is to carry out specialized process containing significant data to these.Big data
Data volume typically more than TB ranks, typically cannot be processed with the computer of separate unit, generally using distributed structure/architecture.It
Characteristic be that distributed data digging is carried out to big data.But it must rely on the distributed treatment of cloud computing, distributed number
According to storehouse and cloud storage, Intel Virtualization Technology.The value dimension of big data is at following 3 aspects：（1）Product is provided to a large amount of consumers
The enterprise of product or service can carry out precision marketing based on big data technology；（2）The medium and small micro- enterprise for doing little and U.S. pattern can be with profit
Made the transition with big data technical service；（3）The traditional forms of enterprises that facing must make the transition under the Internet pressure need to give full play to big data
Potential value.
Either to data assayer or domestic consumer, data visualization is that data analysis tool is most basic to be wanted
Ask.Visualization can intuitively display data, allow data oneself to speak, allow spectators to hear result.And GIS software, can allow number
According to combining with spatial geographical locations, result is more intuitively seen on map, carry out deeper data mining.
The existing software for big data visualized management, mostly professional data analysis software, mostly deploys
The (SuSE) Linux OS of Hadoop running environment, but also to expend high learning cost, longer time cost and for this
Undertake huge expense.
The expert data that is processed to big data, excavates and optimizes is had no lack of in the world processes software, but can be by
Big data is combined with GIS software, big data is processed in GIS software, is excavated, and is combined
The professional GIS software that geographical position is shown is really rare, but also needs visual mode to realize, this GIS at home
Blank field is still belonged in industry.
The content of the invention
Present invention is primarily targeted at providing for the visual management method of big data in a kind of GIS software, to solve
Certainly GIS software analysis efficiency for facing when space-time big data is processed is low in prior art, the problems such as display effect is not good.
In order to solve above-mentioned technical problem, this application provides for the visualized management of big data in a kind of GIS software
Method.The method includes：1）Structure is suitable for the distributed data source of different pieces of information storage mode；2）According to the storage side of data
Formula, input known parameters open corresponding data source, access and read the big data for being stored in server end；3）To what is read
Big data, realizes visual data management operations, including arranges field, creates index, data supplementing, data importing and data
Derive；4）The data that complete will be processed to upload onto the server end, data loading is realized or shared to other people.
Further, the step 1）In data source include HDFS data sources and MongoDB data sources.
Further, the step 3）During big data reads, user can custom-configure, and will meet configuration
The batch data of condition is converted to geographical spatial data.
Further, the step 3）Data management be multi-job operation, show at present just in visual mode
In the multitask for carrying out, Task Progress can be checked, support aligns afoot task to be carried out the operation such as cancelling.
The invention has the beneficial effects as follows：
1. new engine mode, i.e., aforesaid two kinds of data sources are increased：HDFS data sources and MongoDB data sources, user only needs
It is input into corresponding parameter, it is possible to directly read storage big data in the server；During big data reads,
The batch data for meeting configuration condition can be converted to geographical spatial data based on custom-configuring by user, and this process,
Also all it is visual.
2. the present invention is integrated with the reading to big data and management in production domesticization GIS software, and uses visual
The mode of change, increased two kinds of engine modes for being easy to user to understand, both engine modes are also depositing for common big data
Storage mode, in the management of the big data for reading, also use be easy to operation, the processing mode of easy-to-use understanding, and support by
Source data is converted to geographical spatial data.
3. user accesses the big data for being deployed in server end by interactive mode, and final acquisition is suitable for and geography information
The data form of operation, and the information for containing geospatial location is automatically converted to into geographical spatial data；In addition, big data
The mode of distributed storage is all used, and in the present invention, system automatic adaptation bottom physical environment, user need not know
Which platform computer is data are stored in, need to only be input into relevant parameter, and system can read data and show in backstage matching automatically
In front end.
4. the analysis efficiency for facing when space-time big data is processed for common production domesticization GIS software is low, display effect
Not good the problems such as, the present invention will realize visualized management of the production domesticization GIS software to big data, by interactively operation, side
Just intuitively data cluster operated and is managed, realized that direct data analysiss effect shows, help domestic consumer more preferable
Understanding data, data analysiss expert carry out deeper into analysis, Added Management person carries out decision-making.
5. the present invention is based on Spark frameworks, and Scala programming languages build openable in production domesticization desktop GIS software
Distributed data source, user is i.e. by the corresponding parameter such as input address, Instance Name, user name, password, so that it may which acquisition is stored in
The data resource of server end, then arranged by corresponding field parameter, you can be converted to the data form that GIS software can read
（The text of geographic coordinate information will for example be included（CSV）Be converted to spatial point data set）, so as to realize to big data
Efficient visualized management.
The purpose of the present invention is exactly to fill up to the blank of distributed big data management in domestic GIS software, and is to disobey
Bad operating system, is independent of Hadoop running environment, and in visual mode distributed big data is managed, and reduces the operation of user
Difficulty, the service efficiency of significant increase user.
Description of the drawings
Fig. 1 is the flow chart of the visual management method in a kind of GIS software of the invention for big data；
Fig. 2 is the flow chart of reading csv file according to a first embodiment of the present invention.
Specific embodiment
The specific embodiment of the present invention is described in further detail with reference to embodiment：
First embodiment
As shown in Figures 1 and 2, for the visual management method of big data in a kind of GIS software, as illustrated, the method bag
Below step S01 is included to step S03.
Step S01：HDFS distributed data sources are built, the data form of server end storage is csv file.
Step S02：According to the storage mode of data, input known parameters open corresponding data source, access and read and deposit
Store up the big data in server end.
Data are stored in oracle database, when opening in SuperMap GIS softwares, need to be input into server
Address（The server address of storage data）, instance name, another name（The title being displayed in GIS software）, user name, password etc.
Parameter, opens HDFS data sources；
Step S03：To the big data for reading, realize visual data management operations, including arrange field, create index,
Data supplementing, data are imported and data are derived；The data form of server end storage is csv file；To be converted into GIS software
Data with geographic coordinate information are converted to point data collection by discernible data mode；When importing csv file, its first trip
Field, separator etc. all can be setting, the establishment of data directory after importing, also support that the data to importing in batches are carried out
The operations such as additional, data derivation；
A) data that management is read：Based on the bibliographic structure of the mode display data file of directory tree, newly-built, deletion mesh is supported
Record, and to its renaming；By being input into server address, Instance Name, user name, password etc., HDFS data sources are opened；Configuration is read
Association attributes during csv file is taken, the field information in csv file is read and is converted to the discernible field information of software；Read
Taking csv file flow process includes：Association attributes, such as file path, starting row, character code, separation during predefined reading csv file
Symbol etc.；According to predetermined parameter item, association attributes when reading csv file is set；The field structure of predefined csv file；Wound
Index；The field in csv file is read, and is created according to original type；Detect and include geographic coordinate information word
Section, then directly generate point data collection；
B) to the visualized operation of data.The newly-built, additional of data is supported, the interactive operation of client and server is supported, on
Pass and downloading data, and support breakpoint transmission, that supports data imports and exports operation；Access is shown by the way of subwindow
Server directory under file, the content of display includes index, file name, size, Blocksize sizes, the institute of occupancy
The information such as the person of having, packet；
C) the various data management operations for currently carrying out can be checked in task management, is shown at present just in visual mode
In the multitask for carrying out, Task Progress can be checked, support aligns afoot task to be carried out the operation such as cancelling：For HDFS
Data source：First data are set up with specific field information when indexing；Data without index are supported to specify when calculating, analyzing
Field information；And can be by arranging field information matched data collection type.
Step S04：The data that complete will be processed to upload onto the server end, data loading is realized or shared to other people.
Second embodiment
As shown in Figures 1 and 2, for the visual management method of big data in a kind of GIS software, as illustrated, the method bag
Below step S01 is included to step S03.
Step S01：MongoDB distributed data sources are built, the data form of server end storage is csv file.
Step S02：According to the storage mode of data, input known parameters open corresponding data source, access and read and deposit
Store up the big data in server end.
Data are stored in oracle database, when opening in SuperMap GIS softwares, need to be input into server
Address（The server address of storage data）, instance name, another name（The title being displayed in GIS software）, user name, password etc.
Parameter, opens MongoDB data sources；
Step S03：To the big data for reading, realize visual data management operations, including arrange field, create index,
Data supplementing, data are imported and data are derived；The data form of server end storage is csv file；To be converted into GIS software
Data with geographic coordinate information are converted to point data collection by discernible data mode；When importing csv file, its first trip
Field, separator etc. all can be setting, the establishment of data directory after importing, also support that the data to importing in batches are carried out
The operations such as additional, data derivation；
B) data that management is read：Support it is newly-built, deltree, and to its renaming；By being input into server address, example
Name, user name, password etc., open MongoDB data sources；Association attributes during csv file is read in configuration, in reading csv file
Field information and be converted to the discernible field information of software；Reading csv file flow process includes：It is predefined to read csv file
When association attributes, such as file path, starting row, character code, separator；According to predetermined parameter item, it is literary that reading CSV is set
Association attributes during part；The field structure of predefined csv file；Create index；The field in csv file is read, and according to original
There is type to be created；Detect and include geographic coordinate information field, then directly generate point data collection；
B) to the visualized operation of data.The newly-built, additional of data is supported, the interactive operation of client and server is supported, on
Pass and downloading data, and support breakpoint transmission, that supports data imports and exports operation；Access is shown by the way of subwindow
Server directory under file, the content of display includes index, file name, size, Blocksize sizes, the institute of occupancy
The information such as the person of having, packet；
C) the various data management operations for currently carrying out can be checked in task management, is shown at present just in visual mode
In the multitask for carrying out, Task Progress can be checked, support aligns afoot task to be carried out the operation such as cancelling：For
MongoDB data sources：By the table of fixed name（Such as smFieldInfos）, store the field of all tables.By what is arranged
Field information matched data collection type.
Step S04：The data that complete will be processed to upload onto the server end, data loading is realized or shared to other people.
Term is introduced：
Spark：The universal parallel framework increased income, it can be parallel located on large-scale cluster in a kind of reliable and fault-tolerant mode
Reason big data（TB ranks）.By enabling internal memory distributed data collection, it can not only be provided outside interactive inquiry, can also be optimized
Iteration live load.Spark realizes that Scala is used as its application framework by it based on Scala language.Spark and
Scala can be closely integrated, and Scala can easily operate distributed data collection as operating local collection object.
Spark can be used to build large-scale, low latency data analysis application program.Spark provides the Distributed Calculation energy in internal memory
Power, the API DLLs with tetra- kinds of programming languages of Java, Scala, Python, R.
Scala：The programming language of normal form more than one, with characteristics such as object-oriented, functional expression programming, static types, and
With autgmentability, interoperability can be realized with Java and .NET.
HDFS：Hadoop distributed file systems.HDFS is the system of an Error Tolerance, is adapted to be deployed in cheap
On machine.HDFS can provide the data access of high-throughput, the application being especially suitable on large-scale dataset.
The above is only the preferred embodiment of the present invention, it is noted that for one of ordinary skill in the art comes
Say, on the premise of without departing from the technology of the present invention principle, some improvements and modifications can also be made, these improvements and modifications also should
When being considered as within protection scope of the present invention.
Claims (4)
1., for the visual management method of big data in a kind of GIS software, the method includes：1）Build applicable different pieces of information to deposit
The distributed data source of storage mode；2）According to the storage mode of data, input known parameters open corresponding data source, access simultaneously
Reading is stored in the big data of server end；3）To the big data for reading, visual data management operations are realized, including setting
Put field, create index, data supplementing, data importing and data derivation；4）The data that complete will be processed to upload onto the server end,
Realize data loading or shared to other people.
2. for the visual management method of big data in a kind of GIS software according to claim 1, it is characterised in that
The step 1）In data source include HDFS data sources and MongoDB data sources.
3. for the visual management method of big data in a kind of GIS software according to claim 1, it is characterised in that
The step 3）During big data reads, user can custom-configure, and the batch data for meeting configuration condition is turned
It is changed to geographical spatial data.
4. for the visual management method of big data in a kind of GIS software according to claim 1, it is characterised in that
The step 3）Data management be multi-job operation, the multitask for currently carrying out is shown in visual mode, can
To check Task Progress, support aligns afoot task to be carried out the operation such as cancelling.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611182291.6A CN106599241B (en) | 2016-12-20 | 2016-12-20 | Visual management method for big data in GIS software |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611182291.6A CN106599241B (en) | 2016-12-20 | 2016-12-20 | Visual management method for big data in GIS software |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106599241A true CN106599241A (en) | 2017-04-26 |
| CN106599241B CN106599241B (en) | 2020-06-30 |
Family
ID=58599585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611182291.6A Active CN106599241B (en) | 2016-12-20 | 2016-12-20 | Visual management method for big data in GIS software |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106599241B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681450A (en) * | 2018-05-08 | 2018-10-19 | 北京明朝万达科技股份有限公司 | A kind of flow establishment dispositions method and system based on Activiti |
| CN112100123A (en) * | 2020-08-31 | 2020-12-18 | 长江空间信息技术工程有限公司(武汉) | Method for hierarchically displaying large-data-volume CAD (computer-aided design) files at Web front end |
| CN112559453A (en) * | 2020-12-09 | 2021-03-26 | 恒安嘉新（北京）科技股份公司 | Data storage method and device, electronic equipment and storage medium |
| CN112612864A (en) * | 2020-12-28 | 2021-04-06 | 厦门市美亚柏科信息股份有限公司 | Data visualization display method and terminal |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103065205A (en) * | 2012-12-26 | 2013-04-24 | 深圳先进技术研究院 | Three-dimensional intelligent transportation junction passenger flow time-space analysis and prediction system |
| CN104007948A (en) * | 2014-05-23 | 2014-08-27 | 广东威创视讯科技股份有限公司 | Method and device based on distributed computation and visualized display of three-dimensional GIS mass data |
| CN104361110A (en) * | 2014-12-01 | 2015-02-18 | 广东电网有限责任公司清远供电局 | Mass electricity consumption data analysis system as well as real-time calculation method and data mining method |
| KR20160025364A (en) * | 2014-08-27 | 2016-03-08 | 한전케이디엔주식회사 | Smart grids distribute network system |
| CN105512302A (en) * | 2015-12-14 | 2016-04-20 | 浪潮软件股份有限公司 | Distributed GIS (geographic information system) platform system data access and embedding method |
-
2016
- 2016-12-20 CN CN201611182291.6A patent/CN106599241B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103065205A (en) * | 2012-12-26 | 2013-04-24 | 深圳先进技术研究院 | Three-dimensional intelligent transportation junction passenger flow time-space analysis and prediction system |
| CN104007948A (en) * | 2014-05-23 | 2014-08-27 | 广东威创视讯科技股份有限公司 | Method and device based on distributed computation and visualized display of three-dimensional GIS mass data |
| KR20160025364A (en) * | 2014-08-27 | 2016-03-08 | 한전케이디엔주식회사 | Smart grids distribute network system |
| CN104361110A (en) * | 2014-12-01 | 2015-02-18 | 广东电网有限责任公司清远供电局 | Mass electricity consumption data analysis system as well as real-time calculation method and data mining method |
| CN105512302A (en) * | 2015-12-14 | 2016-04-20 | 浪潮软件股份有限公司 | Distributed GIS (geographic information system) platform system data access and embedding method |
Non-Patent Citations (1)
| Title |
|---|
| DUDU131016: ""2个GIS平台软件简介及功能介绍"", 《豆丁网 HTTPS://WWW.DOCIN.COM/P-1026214650.HTML》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681450A (en) * | 2018-05-08 | 2018-10-19 | 北京明朝万达科技股份有限公司 | A kind of flow establishment dispositions method and system based on Activiti |
| CN112100123A (en) * | 2020-08-31 | 2020-12-18 | 长江空间信息技术工程有限公司(武汉) | Method for hierarchically displaying large-data-volume CAD (computer-aided design) files at Web front end |
| CN112100123B (en) * | 2020-08-31 | 2023-06-09 | 长江空间信息技术工程有限公司(武汉) | Method for layering and displaying large-data-volume CAD (computer aided design) files at front end of Web |
| CN112559453A (en) * | 2020-12-09 | 2021-03-26 | 恒安嘉新（北京）科技股份公司 | Data storage method and device, electronic equipment and storage medium |
| CN112612864A (en) * | 2020-12-28 | 2021-04-06 | 厦门市美亚柏科信息股份有限公司 | Data visualization display method and terminal |
| CN112612864B (en) * | 2020-12-28 | 2022-08-16 | 厦门市美亚柏科信息股份有限公司 | Data visualization display method and terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106599241B (en) | 2020-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11797558B2 (en) | Generating data transformation workflows | |
| US11163739B2 (en) | Database table format conversion based on user data access patterns in a networked computing environment | |
| CN106897322B (en) | A kind of access method and device of database and file system | |
| US20230084389A1 (en) | System and method for providing bottom-up aggregation in a multidimensional database environment | |
| US11941016B2 (en) | Using specified performance attributes to configure machine learning pipepline stages for an ETL job | |
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| Karnitis et al. | Migration of relational database to document-oriented database: Structure denormalization and data transformation | |
| US11704331B2 (en) | Dynamic generation of data catalogs for accessing data | |
| US10540383B2 (en) | Automatic ontology generation | |
| CN109964216A (en) | Identify unknown data object | |
| US20190102447A1 (en) | System and method for metadata sandboxing and what-if analysis in a multidimensional database environment | |
| US20110054854A1 (en) | Systems and methods for generating dimensionally altered model objects | |
| US9201700B2 (en) | Provisioning computer resources on a network | |
| CN101901265B (en) | Objectification management system of virtual test data | |
| US8881127B2 (en) | Systems and methods to automatically generate classes from API source code | |
| KR20180121732A (en) | Method and system for analyzing data based on block | |
| EP3173951A1 (en) | Integrated framework for secured data provisioning and management | |
| CN103092906A (en) | Multi-client generic persistence for extension nodes | |
| CN106599241A (en) | Big data visual management method for GIS software | |
| CN107103064A (en) | Data statistical approach and device | |
| CN112860777B (en) | Data processing method, device and equipment | |
| US9373093B2 (en) | Gateway service manager for business object applications | |
| CN111475511A (en) | Data storage method, data access method, data storage device, data access device and data access equipment based on tree structure | |
| Merritt et al. | Big Earth Data: a comprehensive analysis of visualization analytics issues | |
| Parmar et al. | MongoDB as an efficient graph database: An application of document oriented NOSQL database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |