CN102495902A - Method and system for simultaneously realizing ETL (Extract Transform and Load) process of spatial data and attribute data - Google Patents
Method and system for simultaneously realizing ETL (Extract Transform and Load) process of spatial data and attribute data Download PDFInfo
- Publication number
- CN102495902A CN102495902A CN2011104246740A CN201110424674A CN102495902A CN 102495902 A CN102495902 A CN 102495902A CN 2011104246740 A CN2011104246740 A CN 2011104246740A CN 201110424674 A CN201110424674 A CN 201110424674A CN 102495902 A CN102495902 A CN 102495902A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- attribute
- spatial
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Image Generation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and system for simultaneously realizing an ETL (Extract Transform and Load) process of spatial data and attribute data, which relate to the technical field of computerized information processing. In the prior art, an ETL solution of the spatial data is provided but not perfect, and is only limited in extraction, transformation and load of coordinate information, and the combination of the ETL solution and an ETL process of traditional attribute information is not considered. The method comprises the following steps: reading source data from a data source; detecting whether the source data contains graphic element information or not, if yes, transforming the graphic element information which comprises spatial information and style information, and if not, directly transforming into attribute data; and loading the transformed data into a target data source. According to the invention, the ETL process of the attribute data and the ETL process of the spatial data are integrated, so that extraction and transformation between any spatial data and any attribute data are realized.
Description
Technical field
The present invention relates to technical field of computer information processing, relate in particular to the method and system of implementation space data of a kind of while and attribute data ETL process.
Background technology
ETL is the abbreviation of Extract-Transform-Load; It is the process of data pick-up, conversion, loading; Be responsible for cleaning after data such as relation data, flat data file etc. in the data source that will distribute, isomery are drawn into interim middle layer, conversion, integrated; Be loaded at last in target data source or the data warehouse, become the basis of on-line analytical processing, data mining.
Spatial data is meant and is used for the data of the many aspects of position, shape, size and distribution characteristics thereof information of representation space entity, and it can be used for describing the target from real world, and it has characteristics such as location, qualitative, time and spatial relationship.Spatial data is a kind of data of representing the natural world that people depend on for existence with fundamental space data structures such as point, line, surface and entities.
The ETL process of spatial data then be clean after the spatial data (comprising coordinate information) with multi-source, isomery is drawn into interim middle layer, conversion, integrated, be loaded at last in the object space data source of another kind of form.
In existing systems and the technology, major part is to traditional attribute data at present, and it has realized extraction, conversion, the loading procedure of all kinds attribute information between heterogeneous data source (integer, floating type, character type, long text, scale-of-two etc.).It extracts data as record on data source one by one, and through middle treatment scheme, record is then written on the target data source of another kind of form the most at last.
In addition, the ETL solution that has provided spatial data is arranged also in the prior art, but and imperfection, only be confined to extraction, conversion, the loading of coordinate information, and consider to combine with the ETL process of traditional attribute information.The defective of this method is to fail to carry out good binding with traditional attribute data ETL process; Promptly lack changing the mechanism each other between spatial data and the attribute data; Can't solve traditional MIS system (MIS; Management Information System) and the exchanges data between the generalized information system (GIS-Geographic Information System, geographic information system).Therefore, ETL process that can't the implementation space data.Though these methods on the basis of traditional E TL process, expand the traditional data type, introduce new primitive types and be used for storing coordinate information, then by the ETL process of traditional E TL process implementation space data coordinates information.But these methods are only considered the spatial information in the spatial data, but do not consider the style information that comprises in the spatial data, can't in the ETL process, preserve and recover this information, lose through style information after the ETL process.
Summary of the invention
To existing deficiency in the prior art; The present invention proposes a kind of more perfect integration space and the ETL implementation method of attribute data; Its purpose is: the ETL process of implementation space data; Comprise the processing of coordinate information and style information, and under the situation that data source is supported, do not lose after the style information conversion; Realize integration processing, the conversion of attribute data and spatial information.
The present invention solves the problems of the technologies described above the technical scheme that is adopted and describes as follows:
The method of implementation space data of a kind of while and attribute data ETL process may further comprise the steps:
(1) from data source, read source data, described source data is a spatial data;
(2) whether comprise primitive information in the detection resources data, if, then primitive information is carried out conversion process, described primitive information comprises spatial information and style information, otherwise direct converting attribute data;
(3) with the data load after the conversion process in target data source.
Further, in the step (1),, from data source, read source data one by one with the unit of being recorded as.
In the step (2); Directly extract or save as sheerly property data for the attribute field in the source data, promptly attribute data is when realizing the ETL process, according to traditional realization method; Be responsible for from source data, reading attribute field information, and attribute field information is written in the target data source.
Further, in the step (2), when carrying out conversion process, comprise the steps: for the spatial information style information in the source data
1. spatial data information is changed;
2. built-in attribute information is changed;
3. according to the tenability of this Data Format to pattern; Read public style information and privately owned style information to attribute field from source data; And on target data, the attribute field that imports into is retrieved; Therefrom search public style information and privately owned style information, be converted into the style information that target is supported.
Further, step 1. in, the information in the spatial data that reads is decomposed, be decomposed into following a few partial information:
(1) space descriptor: to descriptors such as the type of pel, coordinate systems;
(2) volume coordinate information: geographical location information, i.e. X, Y coordinate;
(3) traditional attribute information: the traditional attribute information that comprises in the spatial data;
(4) space style information: information such as the color of using when drawing pel, filling.
Further again, adopt following mode to handle to these several partial informations in the spatial data that reads:
In the field of primitive types, this field and other attribute field constitute record, follow the ETL process and handle with space descriptor, volume coordinate information stores;
Traditional attribute information is directly handled as attribute field;
The space style information is also handled as attribute field.
Further, to style information, it is divided into following two parts and carries out handled:
(1) public style information: be the total style information that extracts from various data sources, when data source was supported, it can obtain preserving in the ETL in different types of data source process.
Public style information comprises three types:
1. put the object style information: the symbol of point, size, color;
2. line object style information: the symbol of line, width, color;
3. in the face of decent formula information: the symbol of face, whether fill, Fill Color, whether draw sideline, sideline color, sideline width.
(2) privately owned style information: because various spatial data sources have the characteristics of self; Style information outside " public style information " is that the Various types of data source is distinctive; Privately owned style information can only be preserved in the ETL in homogeneous data source process, in the ETL process of different types of data source, will be dropped.
The present invention also provides the system of implementation space data of a kind of while and attribute data ETL process, comprises with lower device:
The source data reading device is used for reading source data from data source, and described source data is a spatial data;
Data Detection and conversion equipment are used for the detection resources data and whether comprise primitive information, if, then primitive information is carried out conversion process, described primitive information comprises spatial information and style information data, otherwise direct converting attribute data;
Data loading device is used for the data load after the conversion process to target data source.
Effect of the present invention is: the present invention with the ETL process conformity of attribute data and spatial data together; Realized the extraction conversion between any spatial data and the attribute data; Addressing space information in a conventional manner is for mis system provides perfect solution with docking of generalized information system; The present invention has also realized preservation and the recovery of style information in the ETL process, and data source can support that design realizes flexibly according to s own situation.In the homogeneous data extraction process, can accomplish that style information does not have to lose.
Description of drawings
Fig. 1 is the schematic diagram of the method for implementation space data of a kind of while and attribute data ETL process;
Fig. 2 is the structural drawing of the system of implementation space data of a kind of while and attribute data ETL process;
Fig. 3 is the process flow diagram of the method for implementation space data of a kind of while and attribute data ETL process.
Embodiment
Present invention is described below in conjunction with accompanying drawing and specific embodiment.
As shown in Figure 1, source data is a spatial data, except comprising traditional attribute information, also comprises spatial information and style information, correspondingly also have attribute information, spatial information and style information in the target data source that will load.Adopt method of the present invention, can the ETL process conformity of attribute data and spatial data (spatial information and style information data) be in the same place, realized extraction, conversion and loading between any spatial data and the attribute data.
As shown in Figure 2, the system of implementation space data of a kind of while and attribute data ETL process comprises with lower device:
Source data reading device 11 is used for reading source data from data source, and described source data is a spatial data;
Data Detection and conversion equipment 12 are used for the detection resources data and whether comprise primitive information, if, then primitive information is carried out conversion process, described primitive information comprises spatial information and style information data, otherwise direct converting attribute data;
In the present embodiment, comprise following two processing modules in described Data Detection and the conversion equipment:
(1) the attribute information processing module 121: be used for carrying out conversion process to traditional attribute information;
(2) the spatial information processing module 122: be used for carrying out conversion process to spatial information;
Described spatial information processing module comprises following four types of submodules:
1. submodule 1, be used for spatial information is transformed to spatial information, its realization be the conversion of spatial data;
2. submodule 2; Be used for spatial information is transformed to attribute information; Its with volume coordinate information with stored in form such as character strings in attribute field, simultaneously the style information in the spatial data also can become traditional attribute field value as requested, has promptly realized the process with the spatial information attributed;
3. submodule 3, are used for attribute information is transformed to spatial information, obtain volume coordinate information according to certain format analysis in its dependency field, promptly realized the process with the attribute information spatialization.
4. submodule 4, are used for attribute information is transformed to attribute information.
As shown in Figure 3, the method for implementation space data of a kind of while and attribute data ETL process may further comprise the steps:
(1) from data source, read source data, described source data is a spatial data;
(2) whether comprise primitive information in the detection resources data, if, then primitive information is carried out conversion process, described primitive information comprises spatial information and style information, otherwise direct converting attribute data;
(3) with the data load after the conversion process in target data source.
The method of aforesaid implementation space data of a kind of while and attribute data ETL process in the step (1), with the unit of being recorded as, reads source data one by one from data source.
In the present embodiment; Directly extract or save as sheerly property data by the attribute information processing module for the attribute field in the source data; Be that attribute data is when realizing the ETL process; According to traditional realization method, be responsible for from source data, reading attribute field information, and attribute field information is written in the target data source;
In the present embodiment,, then carry out conversion process, comprise the steps: by the spatial information processing module for spatial information in the source data and style information
1. spatial information is read and writes;
2. built-in attribute information is read and writes;
3. according to the tenability of this Data Format to pattern; Read public style information and privately owned style information to attribute field from source data; And on target data, the attribute field that imports into is retrieved; Therefrom search public style information and privately owned style information, be converted into the style information that target is supported, and be written in the target data source.
Step 1. in, for the implementation space data ETL process of (comprising spatial information and style information), in the present embodiment information in the spatial data is decomposed, be decomposed into following a few partial information:
(1) space descriptor: to descriptors such as the type of pel, coordinate systems;
(2) volume coordinate information: geographical location information, i.e. X, Y coordinate;
(3) traditional attribute information: the traditional attribute information that comprises in the spatial data;
(4) space style information: information such as the color of using when drawing pel, filling.
Further, following mode is adopted in the processing of these several partial informations in the spatial data:
In the field of primitive types, this field and other attribute field constitute record, follow the ETL process and handle with space descriptor, volume coordinate information stores;
Traditional attribute information is directly handled as attribute field;
The space style information is also handled as attribute field.
To style information, the present invention further is divided into two parts with it:
(1) public style information: be the total style information that extracts from various data sources, when data source was supported, it can obtain preserving in the ETL in different types of data source process.
Public style information comprises three types:
1. put the object style information: the symbol of point, size, color;
2. line object style information: the symbol of line, width, color;
3. in the face of decent formula information: the symbol of face, whether fill, Fill Color, whether draw sideline, sideline color, sideline width.
(2) privately owned style information: because various spatial data sources have the characteristics of self; Style information outside " public style information " is that the Various types of data source is distinctive; Privately owned style information can only be preserved in the ETL in homogeneous data source process, in the ETL process of different types of data source, will be dropped.
To sum up, adopt method and system of the present invention, source data realized that following ETL carries out situation:
(1) attribute data is drawn into attribute data: this is that traditional ETL extracts mode
(2) spatial data is drawn into spatial data: spatial data is drawn in the spatial data, has not only kept coordinate information, also can be according to the ability of target data source, and storage, reduction style information.If homogeneous data extracts, can accomplish that style information does not have to lose.
(3) attribute data is drawn into spatial data: attribute data is drawn in the process of spatial data, needs the processor through the attribute space function, obtains volume coordinate information according to certain format analysis in the dependency field.Simultaneously can increase the pattern field, the pattern of the spatial data that control generates to attribute data.
(4) spatial data is drawn into attribute data: spatial data is drawn in the process of attribute data, needs the processor through the space attribute function, with volume coordinate information with stored in form such as character strings in attribute field.Style information in the spatial data also can become traditional attribute field value as requested simultaneously, and supply is used with system.
(5) for the processing of style information; Also having a kind of scheme is that it is stored on the additional information of space pel field; This scheme also can realize the storage and the recovery of style information in the ETL of spatial data process; But but can't dock with traditional attribute data source, style information can't be stored on the common property field, only if the attribute data source is realized again.
It will be understood by those skilled in the art that top specific descriptions just in order to explain the object of the invention, are not to be used to limit the present invention.Protection scope of the present invention is limited claim and equivalent thereof.
Claims (10)
1. method of implementation space data and attribute data ETL process simultaneously may further comprise the steps:
(1) from data source, read source data, described source data is a spatial data;
(2) whether comprise primitive information in the detection resources data, if, then primitive information is carried out conversion process, described primitive information comprises spatial information and style information, otherwise direct converting attribute data;
(3) with the data load after the conversion process in target data source.
2. the method for implementation space data of a kind of while as claimed in claim 1 and attribute data ETL process is characterized in that: in the step (1), with the unit of being recorded as, from data source, read source data one by one.
3. the method for implementation space data of a kind of while as claimed in claim 1 and attribute data ETL process; It is characterized in that: in the step (2); Directly extract or save as sheerly property data for the attribute field in the source data, promptly attribute data is when realizing the ETL process, according to traditional realization method; Be responsible for from source data, reading attribute field information, and attribute field information is written in the target data source.
4. like the method for arbitrary described implementation space data of a kind of while of claim 1 to 3 and attribute data ETL process, it is characterized in that: in the step (2), when carrying out conversion process, comprise the steps: for the spatial information in the source data and style information
1. spatial data information is changed;
2. built-in attribute information is changed;
3. according to the tenability of this Data Format to pattern; Read public style information and privately owned style information to attribute field from source data; And on target data, the attribute field that imports into is retrieved; Therefrom search public style information and privately owned style information, be converted into the style information that target is supported.
5. the method for implementation space data of a kind of while as claimed in claim 4 and attribute data ETL process is characterized in that: step 1. in, the information in the spatial data that reads is decomposed, be decomposed into following a few partial information:
(1) space descriptor: the type, the coordinate system that comprise pel;
(2) volume coordinate information: geographical location information, i.e. X, Y coordinate;
(3) traditional attribute information: the traditional attribute information that comprises in the spatial data;
(4) space style information: comprise the color, the filling information that use when drawing pel.
6. the method for implementation space data of a kind of while as claimed in claim 5 and attribute data ETL process is characterized in that, adopts following mode to handle to the information in the spatial data that reads:
In the field of primitive types, this field and other attribute field constitute record, follow the ETL process and handle with space descriptor, volume coordinate information stores;
Traditional attribute information is directly handled as attribute field;
The space style information is also handled as attribute field.
7. the method for implementation space data of a kind of while as claimed in claim 6 and attribute data ETL process is characterized in that, to style information, it is divided into following two parts and carries out handled:
(1) public style information: be the total style information that extracts from various data sources, when data source was supported, it can obtain preserving in the ETL in different types of data source process;
Public style information comprises three types:
1. put the object style information: the symbol of point, size, color;
2. line object style information: the symbol of line, width, color;
3. in the face of decent formula information: the symbol of face, whether fill, Fill Color, whether draw sideline, sideline color, sideline width;
(2) privately owned style information: because various spatial data sources have the characteristics of self; Style information outside " public style information " is that the Various types of data source is distinctive; Privately owned style information can only be preserved in the ETL in homogeneous data source process, in the ETL process of different types of data source, will be dropped.
8. system of implementation space data and attribute data ETL process simultaneously comprises with lower device:
The source data reading device is used for reading source data from data source, and described source data is a spatial data;
Data Detection and conversion equipment are used for the detection resources data and whether comprise primitive information, if, then primitive information is carried out conversion process, described primitive information comprises spatial information and style information, otherwise direct converting attribute data;
Data loading device is used for the data load after the conversion process to target data source.
9. the system of implementation space data of a kind of while as claimed in claim 8 and attribute data ETL process is characterized in that, comprises following two processing modules in described Data Detection and the conversion equipment:
(1) attribute information processing module: be used for carrying out conversion process to traditional attribute information;
(2) spatial information processing module: be used for carrying out conversion process to spatial information.
10. the system of implementation space data of a kind of while as claimed in claim 9 and attribute data ETL process is characterized in that, described spatial information processing module is divided into following four types of submodules:
1. submodule 1, be used for spatial information is transformed to spatial information, its realization be the conversion of spatial data;
2. submodule 2; Be used for spatial information is transformed to attribute information; Its with volume coordinate information with stored in form such as character strings in attribute field, simultaneously the style information in the spatial data also can become traditional attribute field value as requested, has promptly realized the process with the spatial information attributed;
3. submodule 3, are used for attribute information is transformed to spatial information, obtain volume coordinate information according to certain format analysis in its dependency field, promptly realized the process with the attribute information spatialization;
4. submodule 4, are used for attribute information is transformed to attribute information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110424674 CN102495902B (en) | 2011-12-16 | 2011-12-16 | Method and system for simultaneously realizing ETL (Extract Transform and Load) process of spatial data and attribute data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110424674 CN102495902B (en) | 2011-12-16 | 2011-12-16 | Method and system for simultaneously realizing ETL (Extract Transform and Load) process of spatial data and attribute data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102495902A true CN102495902A (en) | 2012-06-13 |
| CN102495902B CN102495902B (en) | 2013-07-24 |
Family
ID=46187727
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110424674 Active CN102495902B (en) | 2011-12-16 | 2011-12-16 | Method and system for simultaneously realizing ETL (Extract Transform and Load) process of spatial data and attribute data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102495902B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104794252A (en) * | 2014-01-17 | 2015-07-22 | 中国石油集团工程设计有限责任公司 | Three-dimensional model data processing method and electronic terminal |
| CN112181961A (en) * | 2020-09-25 | 2021-01-05 | 杭州安恒信息技术股份有限公司 | Method, system and related device for cleaning network data |
| CN113868280A (en) * | 2021-11-25 | 2021-12-31 | 芯和半导体科技(上海)有限公司 | Parameterized unit data updating method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101196797A (en) * | 2007-12-07 | 2008-06-11 | 华中科技大学 | Memory system data arrangement and commutation method |
| CN101609465A (en) * | 2009-07-16 | 2009-12-23 | 浙江大学 | A kind of fast conversion method of space vector data |
| CN101917449A (en) * | 2010-09-01 | 2010-12-15 | 中国地质大学（武汉） | Three-dimensional spatial data transmission-oriented application layer communication method |
-
2011
- 2011-12-16 CN CN 201110424674 patent/CN102495902B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101196797A (en) * | 2007-12-07 | 2008-06-11 | 华中科技大学 | Memory system data arrangement and commutation method |
| CN101609465A (en) * | 2009-07-16 | 2009-12-23 | 浙江大学 | A kind of fast conversion method of space vector data |
| CN101917449A (en) * | 2010-09-01 | 2010-12-15 | 中国地质大学（武汉） | Three-dimensional spatial data transmission-oriented application layer communication method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104794252A (en) * | 2014-01-17 | 2015-07-22 | 中国石油集团工程设计有限责任公司 | Three-dimensional model data processing method and electronic terminal |
| CN112181961A (en) * | 2020-09-25 | 2021-01-05 | 杭州安恒信息技术股份有限公司 | Method, system and related device for cleaning network data |
| CN113868280A (en) * | 2021-11-25 | 2021-12-31 | 芯和半导体科技(上海)有限公司 | Parameterized unit data updating method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102495902B (en) | 2013-07-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105283855B (en) | A kind of addressing method and device | |
| CN104662583A (en) | Gpu memory buffer pre-fetch and pre-back signaling to avoid page-fault | |
| CN102831069B (en) | Internal memory processing method, memory management equipment | |
| CN101398823B (en) | Method and system for implementing remote storage by virtual file systems technology | |
| CN101520797B (en) | High-speed concurrent access method for power system large data files across platform | |
| CN104331545B (en) | A kind of implementation method of the grid automation GIS electrical layers based on CIM/G | |
| CN105787012B (en) | A kind of method and storage system improving storage system processing small documents | |
| CN104461390A (en) | Method and device for writing data into imbricate magnetic recording SMR hard disk | |
| CN102158349A (en) | Log management device and method thereof | |
| CN102810116B (en) | Automatic routing and load balancing method and system based on database connection | |
| CN110597900B (en) | Method for generating vector slice by GDB data in real time according to needs | |
| CN104603834A (en) | Methods and systems for multimedia data processing | |
| CN105528460A (en) | Establishing method of tile pyramid model and tile reading method | |
| CN100449545C (en) | Method and system for accessing sector data | |
| CN103425785A (en) | Data storage system and user data storage and reading method thereof | |
| CN102495902B (en) | Method and system for simultaneously realizing ETL (Extract Transform and Load) process of spatial data and attribute data | |
| CN102622476A (en) | CAD (Computer Aided Design) drawing integration system for field of building amount of computation | |
| CN103051671A (en) | Repeating data deletion method for cluster file system | |
| CN106682110A (en) | Video file storing and managing system and method based on Hash grid index | |
| CN103353866A (en) | Three-dimensional model file format conversion method supporting XNA technology | |
| CN104985939A (en) | Laser marking machine control method and laser marking machine | |
| CN107066562A (en) | A kind of storage method of satellite remote-sensing image data | |
| CN104699826B (en) | A kind of the pyramid laminar storage method and Spatial Database Systems of image data | |
| CN104574275B (en) | A kind of method for merging textures during modeling rendering | |
| CN104166715A (en) | Vxworks platform electronic chart engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |