CN105868326A - Pipeline data storage method - Google Patents

Abstract

Disclosed is a pipeline data storage method. The method is characterized by comprising the steps that 1, pipeline data is classified according to pipeline categories and stored into a GIS spatial data format; 2, the pipeline data of the GIS spatial data format in the step 1 is converted into a JSON data format; 3, Hadoop is utilized for guiding the JSON data format in the step 2 into a data storage catalogue. According to the pipeline data storage method, the problem that time is consumed when a large amount of pipeline data is stored and copied is effectively avoided, and data integrity is well protected.

Description

A kind of pipeline date storage method

Technical field

The present invention relates to pipeline field of data storage, particularly relate to a kind of pipeline date storage method.

Background technology

Underground Pipeline General Survey have accumulated substantial amounts of Underground Pipeline Data, in all parts of the country all at the pipelines GIS building this locality (Geographic Information System) system.With the construction of these pipeline systems, relevant to pipeline operation security Dynamic data, have every day hundreds and thousands of the even record of hundreds of thousands bar constantly producing, but current pipeline data Storage mainly uses the relational databases such as MySQL, Oracle, SQL Server.Excavate from the data of magnanimity fast and effectively Go out useful information to become, allow people increasingly feel helpless.

Summary of the invention

In order to solve above-mentioned technical problem, the present invention proposes a kind of pipeline date storage method, and it effectively prevent large quantities of The storage of buret line data, the time-consuming problem of copy, the most preferably protect data integrity.

To achieve these goals, the scheme that the present invention uses is:

A kind of pipeline date storage method, including step:

S1, pipeline data are carried out classification according to pipeline classification store into GIS spatial data form；

S2, the pipeline data of the GIS spatial data form described in step S1 are converted into JSON data form；

S3, utilize Hadoop the JSON data form described in step S2 is imported to data storage catalogue under.

Described step S1 includes step:

S11, the relevant configuration table in reading field data storehouse, judge whether deposit in field data storehouse according to pipeline group coding In tables of data and the type of tables of data of corresponding pipeline,

S12, if there is no continuing to search for next pipeline type；

S13, if there is a table, then in a table, gradually read record, the X-coordinate and the Y coordinate that obtain every record are raw Become Point type；

S14, if there is line table, then according to starting point pipeline period and terminal pipeline period, at the pipe of corresponding pipeline type Point field data table is searched X-coordinate and the Y coordinate of corresponding record, generates line sections point and end of pipe respectively, according to pipeline Duan Qidian and end of pipe generate line sections PolyLine type；

All pipeline type in S15, traversal allocation list repeat step S12, and S13, S14 generate all of pipe point and pipeline Section spatial data table.

Described step S1 includes step:

S01, collection pipeline data, set up pipeline CAD database；

S02, pipeline CAD database is converted to GIS spatial data form.

Described step S01 includes step:

S011, utilizes the cloud data of laser scanner locality underground pipelines network；

S012, is forwarded to the cloud data obtained in step S011 in same coordinate system by Registration of Measuring Data；

S013, rejects the noise data in the cloud data under same coordinate system；

S014, takes out dilute process to the cloud data rejecting noise data in step S013；

S015, the cloud data after processing according to step S014 sets up pipeline surface model；

S016, the pipeline surface modeling rendering pipeline model generated according to step S015；

S017, the pipeline model drawn according to step S016, extracts pipeline information；

S018, the pipeline information extracted according to step S017, reads the system configuration parameter pipe point to extracting, pipeline data Check, and will check that result is according to type of error list；

S019, the pipeline model rendering pipeline 2 d plane picture drawn according to step S016, extracts according to step S018 Pipeline information mark pipeline kind, material and caliber information, finally give pipeline result map；

S0110, by pipeline result map, the graphics of pipeline model, pipeline 2 d plane picture preserves to pipeline CAD database In.

Described step S02 includes step:

S021, the layering of pipeline cad data is converted to GIS spatial data, and records corresponding attribute item；

S022, the attribute information in pipeline cad data is assigned to corresponding GIS spatial data, as GIS spatial data Attribute information.

Described step S2 includes step:

S21, travel through all of spatial data table；

S22, judge data table types, generate JSON format module according to data table types and lay equal stress on entitled with spatial data table Identical title；

S23, open template at file the beginning part with the form of key-value pair: " key: value " adds Spatial data types, reference Coordinate system, often group key-value pair separates with ", " respectively；

S24, in field groups add list of fields, each field includes title, type, length three groups of key-value pairs, often groups Key-value pair separates with ", " respectively, and three groups of key assignments are placed in " { } ", and adjacent field is split with ", "；All field values wrap with " [] " Including, all of field forms key-value pair with field value respectively；

S25, adding data record in data group, all records and data group form key-value pair, and all records are with " [] " Including, ", " segmentation between every record；Every record includes set of properties and two subgroups of geometry, with key in set of properties Value to form deposit the attribute list of record, each attribute uses: attribute: value, and key-value pair form is used between adjacent key-value pair ", " separately, all of attribute key-value pair leaves in " { } "；Geometry is deposited the JSON form number of geometry type According to；

S26, traversal current spatial tables of data repeat S24, S25 step, preserve all of data record, finally at file Beginning and end adds " { ", " } " respectively, then preserves；

S27, in spatial database according to step S22, S23, S24, S25, S26 process all of spatial data table.

Described step S3 includes: step

The JSON data of formatting are imported under catalogue by the S31 ,-put order carried by Hadoop.

-the put that Hadoop in described step S31 carries orders the operation that works in a parallel fashion.

Described step S011 includes step:

S111, draws piping lane trend on topographic map, region to be measured is divided into several grids；

S112, includes a survey station and at least three target point in each grid；

A113, to all grids in scope to be measured, sets up three-dimensional laser scanner one by one；

S114, in each grid, sets up three-dimensional cartesian coordinate system with laser scanner for initial point: wherein, and X-axis is laterally being swept Retouching in face, Y-axis is vertical with X-axis in transversal scanning face, and Z axis is vertical with transversal scanning face；Utilize laser scanner measurement laser A survey station in the grid of scanner place and the coordinate of three target points；

S115, repetition step S112 is to step S114, until all grids are the most measured complete.

Registration of Measuring Data in described step S012 utilizes boolean's sand Seven-parameter to carry out Registration of Measuring Data；In described step S013 Macroscopic examination method or curve inspection technique or string high differentiation is utilized to carry out noise rejecting；Described step S014 takes out dilute process Dilute distance of taking out be 5cm, described step S015 is set up pipeline surface model utilize method be Di Luoni triangle terrain model.

The invention have the benefit that and take three-dimensional laser scanning technique to gather complicated piping lane data, improve office certainly Dynamicization degree, improves operating efficiency, alleviates staff's homework burden；Carry out pipeline modeling under three-dimensional environment, improve Data identification degree, reduces the frequency of mistake；Pipeline information data are automatically extracted by configurable pipeline tables of data, Alleviate the complexity that staff's data process；Improved by the configuration inspection contents of a project and check the degree of accuracy, disposably examine Look into all the elements, it is to avoid rechecking, save project cost.

Cad data is converted directly into GIS spatial data, relative to directly setting up GIS spatial data, save the time with Operation.Cad data itself is also a database simultaneously, is also a database of demand in practice, it means that the party The process of formula, it is individual very succinct efficient for being converted to GIS spatial data by data in the technology set up CAD database 's.

Utilize compression storage pipeline data, provide a kind of new thinking for pipeline data storage technology, to greatest extent Reduce the burden of file system, the lookup so not only increasing file, the efficiency copying, shifting, also can be effectively improved calculating Machine or various hand-held embedded removable standby storage efficiency, be effectively increased the operational efficiency of various equipment.

Accompanying drawing explanation

Fig. 1 pipeline compression storing data flow chart；

Fig. 2 pipeline tables of data institutional framework；

Fig. 3 pipe space product process；

Fig. 4 JSON data format flow process；

Fig. 5 JSON formatted data organization chart.

Detailed description of the invention

In order to be better understood by technical scheme, the invention will be further described below in conjunction with the accompanying drawings.

(1) SO1 gathers pipeline data, sets up pipeline CAD database.

S011, utilizes the cloud data of laser scanner locality underground pipelines network；

The high accuracy cloud data with the image sense of reality is obtained by airborne lidar instrument or terrestrial Laser scanner, Cloud data is the recovery of the full-size(d) of actual object, is to carry out object present situation the most completely, the most finely and efficiently at present The means of archives preservation.Each scanning movement point data be placed on independent coordinate system centered by instrument (with instrument as initial point, X-axis is in transversal scanning face, and Y-axis is vertical with X-axis in transversal scanning face, and Z axis is vertical with transversal scanning face).Impact point P coordinate Formula: Xp=Scos θ Cos φ；Yp=Scos θ sin φ；Zp=Ssin θ.Wherein, S is the distance of measuring point and scanner；φ is The transversal scanning angular observation of laser pulse；Regulation of longitudinal angle scanning observation θ；P for observation and three coordinates (Xp, Yp, Zp)。

Needing to carry out collection site actual prospecting before data acquisition, understand coverage of survey area, piping lane moves towards, piping lane class Type, piping lane width information, topographic map is drawn piping lane trend, in the range of measuring, carries out on laterally and longitudinally respectively The row and column of spacing divides, and wherein line space need not be equal to column pitch, defines two adjacent row and two adjacent row X-shapes The region become is grid, includes a survey station and at least 3 target points in each grid, and every a line measures band as one, protects Demonstrate,prove the data that each scanning movement finally obtains and can represent complete measured zone.Survey station is set in grid and numbers, coding rule Employing: survey district's numbering (XXXX)+line number (XX)+numbering (XX), on measurement band center line, survey station is set, to reduce survey station as far as possible Quantity.Arranging target in each grid, target is arranged on measurement band center line and both sides, and adjacent 3 targets do not exist On same straight line, and to target Unified number, naming rule uses: survey district's numbering (XXXX)+line number (XX)+row number (XX)+volume Number (XX).Carrying out data collection and analysis according to the actual situation of making an on-the-spot survey, data collection includes grasping the scope surveying district, piping lane Trend, piping lane length, piping lane type, the kind of pipeline model, the function of equipment, the various duties of equipment, the behaviour of equipment Make mode, collect the survey district topographic map of corresponding engineer's scale according to pipeline result map engineer's scale, collect survey district striograph clearly.

Carried out surveying district by the three-dimensional laser scanner that sets up one by one on the survey station pre-set of actual investigation Scanning, obtains and surveys district's cloud data.Control Target Center and survey station point installation GPS or obtain control by total powerstation (x, y, z), record preserves the three-dimensional coordinate of Target Center processed and survey station point, provides control point information for Registration of Measuring Data.The most right At scanning survey station region gut line turning point, pipeline gland, at cross pipe-line and the hypsography ground more than 20 centimetres Taking pictures in side, photo naming rule is: and survey station numbering (such as: XXXXX2)+type (turning point, 01；Gland, 02；Intersection 03；Rise and fall, 04；Other, 11)+numbering (XXX1).

S012, is forwarded to the cloud data obtained in step S11 in same coordinate system by Registration of Measuring Data.Arbitrarily The cloud data that adjacent two survey station spot scans obtain, not through overmatching, but has more than at least 3 in each grid Target point, carries out Coordinate Conversion according to these identical reference points, unified to same coordinate system.This place uses boolean Sha seven Parametric technique carries out Registration of Measuring Data, as shown in Figure 10, three coordinate translation amounts (△ X, △ Y, △ Z), i.e. two space coordinates The origin of coordinates between coordinate difference；The anglec of rotation (the ω of three reference axis_x, ω_y, ω_z), by rotating three seats in order Parameter, to specified angle, can make the X1Y1Z1 axle of two rectangular coordinate system in space coincide together；Scale factor m, i.e. two The lenth ratio of same section of straight line in space coordinates, it is achieved the ratio conversion of yardstick.

Registration of Measuring Data process:

1. in the range of each grid, seven parameters are calculated by known 3 target control points according to below equation:

$\left[\begin{array}{c}{X}_2\\ Y_2\\ Z_2\end{array}\right]=(1+m)R_1\left({\mathrm{\ω}}_x\right)R_2\left({\mathrm{\ω}}_y\right)R_3\left({\mathrm{\ω}}_z\right)\left[\begin{array}{c}{X}_1\\ Y_1\\ Z_1\end{array}\right]+\left[\begin{array}{c}\mathrm{\Δ}X\\ \mathrm{\Δ}Y\\ \mathrm{\Δ}Z\end{array}\right]---\left(1\right)$

Wherein

$R_1\left({\mathrm{\ω}}_x\right)=\left[\begin{array}{ccc}1& 0& 0\\ 0& {\mathrm{cos\ω}}_x& {\mathrm{sin\ω}}_x\\ 0& -{\mathrm{sin\ω}}_x& {\mathrm{cos\ω}}_x\end{array}\right]---\left(2\right)$

$R_2\left({\mathrm{\ω}}_y\right)=\left[\begin{array}{ccc}{\mathrm{cos\ω}}_y& 0& -{\mathrm{sin\ω}}_y\\ 0& 1& 0\\ {\mathrm{sin\ω}}_y& 0& {\mathrm{cos\ω}}_y\end{array}\right]---\left(3\right)$

$R_3\left({\mathrm{\ω}}_z\right)=\left[\begin{array}{ccc}{\mathrm{cos\ω}}_z& {\mathrm{sin\ω}}_z& 0\\ -{\mathrm{sin\ω}}_z& {\mathrm{cos\ω}}_z& 0\\ 0& 0& 1\end{array}\right]---\left(4\right)$

(2) (3) (4) are substituted into (1), due to generally ω_x, ω_y,ω_zFor slight rotation angles, Ke Yiqu:

$\left\{\begin{array}{c}{\mathrm{cos\ω}}_x={\mathrm{cos\ω}}_y={\mathrm{cos\ω}}_z=1\\ {\mathrm{sin\ω}}_x={\mathrm{\ω}}_x,{\mathrm{sin\ω}}_y={\mathrm{\ω}}_y,{\mathrm{sin\ω}}_z={\mathrm{\ω}}_z\\ {\mathrm{sin\ω}}_x{\mathrm{sin\ω}}_y={\mathrm{sin\ω}}_x{\mathrm{sin\ω}}_z{\mathrm{sin\ω}}_z={\mathrm{sin\ω}}_y{\mathrm{sin\ω}}_z=0\end{array}\right.---\left(5\right)$

Therefore the condition having top simplifies (1), can obtain formula below:

$\left[\begin{array}{c}{X}_2\\ Y_2\\ Z_2\end{array}\right]=(1+m)\left[\begin{array}{ccc}1& {\mathrm{\ω}}_z& -{\mathrm{\ω}}_y\\ -{\mathrm{\ω}}_z& 1& {\mathrm{\ω}}_x\\ {\mathrm{\ω}}_y& -{\mathrm{\ω}}_x& 1\end{array}\right]\left[\begin{array}{c}{X}_1\\ Y_1\\ Z_1\end{array}\right]+\left[\begin{array}{c}\mathrm{\Δ}X\\ \mathrm{\Δ}Y\\ \mathrm{\Δ}Z\end{array}\right]---\left(6\right)$

2. calculate and survey seven mean parameters in district, the average of seven parameters of the most each grid computing, prevent because of local error Cause precision uneven,

3. substitute into formula (6) according to above seven parameters calculated, calculate each survey station spot scan cloud data in target New coordinate in coordinate system, thus realize Registration of Measuring Data.

S013, rejects the noise data in the cloud data under same coordinate system.Noise is deleted: at non-contact three-dimensional During scanning survey, affected, inevitably by factors such as metering system, object being measured material character, external interference The biggest noise spot of error and distorted spots can be produced.The most in data handling, search noise spot and distorted spots that may be present, It is processed.

Noise is deleted and is divided into 3 kinds of methods: 1. macroscopic examination method: by graphic display terminal, is with the naked eye directly present in screen Acnode on curtain is deleted.2. curve inspection technique: by the first and last data point in cross section, obtain with least square fitting SPL, order of a curve time can determine that according to the shape of curved section usually 3-4 rank calculate intermediate data the most respectively Point pi, to distance e of SPL, if e is more than or equal to ε (ε is given franchise), then thinks that pi is bad point, should give and pick Remove；3. action difference method: as it is shown on figure 3, connect 2 points before and after checkpoint, calculates intermediate data points pi distance e to string, as Really e >=ε (ε is given franchise), then think that pi is bad point, should give rejecting.

Noise point delete step, (1) opens cloud data and surveys district's striograph, using striograph as reference, by naked eyes pair Than observing in cloud data not isolated point in the range of piping lane, delete, (2) automatically calculated a traversal point cloud number by action Delete according to noise successively, i.e. checkpoint in cloud data is gradually connected 2 points before and after checkpoint, calculate current check point P and arrive Distance d of 2 lines front and back, if d >=ε (maximum limit that ε is given is poor), then thinks that P is bad point, should give rejecting.

S014, takes out dilute process to the cloud data rejecting noise data in step S013；Data reduction processes: some cloud In scan data, the interval put and put is less, only millimeter rank, and quantity is relatively big, and speed is relatively slow in data handling, causes not Necessary trouble, in order to improve data processing speed, simplifies cloud data on the premise of meeting mapping precision requirement Process, improve operating efficiency.

Cloud data is the set of the coordinate points much with X, Y, Z coordinate, and the coordinate data difference of adjacent 2 is fixing Numerical value only has millimeter rank.Being all unnecessary as these points of pipeline data, so cloud data to be taken out dilute process, reducing Data volume, processing speed at raising.Cloud data setting is taken out dilute interval and is counted or take out dilute distance (acquiescence takes out dilute distance 5cm), Resolve mean square error of coordinate I class precision according to detection point pipeline in " urban underground pipeline exploration code " to require as ± (5+0.02h) Cm, is defaulted as 5cm on the premise of meeting data precision, and wherein h is pipeline buried depth, as h≤70cm, gives tacit consent to 70cm.

It is as follows that data take out dilute process: is 1. opening cloud data txt file；2. in data processing module arrange take out dilute away from From or arrange and take out dilute n of counting (count (n)=take out dilute distance (Δ l)/some cloud interval (d)) in interval；3. according to take out dilute distance or It is spaced the n that counts and starts to read to the last one article from the 1st of data file the article of record, gradually delete the 1st article and recorded (n+1)th article Data between record, the like, a to the last record.

S015, the cloud data after processing according to step S14 sets up pipeline surface model.Pipeline surface models: pass through Di Sieve Buddhist nun's triangulation sets up TIN (Triangulated Irregular Network) pipeline surface model.Take out dilute through data The minimum interval Δ l of adjacent 2 in rear cloud data, the largest interval between adjacent 2 is (√ 2) Δ l, is setting up Di Luoni During the triangulation network, if the distance of adjacent 2 is more than (√ 2) Δ l, then abandon building the triangulation network, to prevent the point in adjacent lines Connecting and composing the surface model of mistake, if being smaller than (√ 2) Δ l according to actual conditions adjacent lines, reducing the most accordingly Take out dilute spacing so that maximum is taken out dilute distance and is not more than the minimum of a value of adjacent lines spacing.

Triangulation network establishment step: 1. arbitrarily look for a bit in the discrete point gathered, then looks up the point nearest away from this point, As initial baseline after connection.2. Delaunay rule is used to search thirdly on the right side of initial baseline, i.e. right at initial baseline The discrete point of side is searched the point the shortest away from this parallax range, as thirdly.3. Delaunay triangle is generated, then with triangle Two new limits of shape (from baseline starting point to thirdly and thirdly to baseline terminating point) are as new baseline.4. step is repeated The most 2., 3. until all of Baseline Survey is complete.

S016, the pipeline surface modeling rendering pipeline model generated according to step S015；System shape library provides more than 25 Kind shape library, including W.N flange, slip-on welding flange, screwed flange, loose flange, blind plate, blind flange, equal tee, reducing, Threeway, four-way, six logical, Concentric Reducers, eccentric reducer, pipe cap, flange, end socket, Guan Tai, plug, 45 ° of elbows, 90 ° curved Head, reducing bend, round tube, side straight tube, bend pipe, lamp stand.With graphical interfaces and parameterized design, rendering pipeline model, can Position and the position of the mouth of pipe with amendment equipment, it is also possible to amendment equipment size, material, position.When rendering pipeline 3D model Can be switched at any time two dimension view inspection select model pipeline, pipe fitting the most correct.

Pipeline model editing process: 1. select one from pipeline surface model, as pipeline to be edited；2. at figure Storehouse manually selects the model needing to add；3., in data edition window, pipeline model specific characteristic point is (typically surface modes The middle peak at type two ends is as the beginning and end of model)；4. existing striograph, photo, the inspection of construction drawing data are opened Look into characteristic point the most correct, if incorrect, then delete, repaint pipe fitting；5. select the pipeline model of input, select amendment Attribute, in parameter arranges forms, revises model parameter (pipeline kind, caliber, material, adjunct, color, wherein pipeline kind Class is required item)；6. the pipeline model of editor is preserved.

S017, the pipeline model drawn according to step S16, extracts pipeline information；According to surveying the pipeline kind that district is comprised System configuration item is set, extracts pipeline information according to configuration item pipeline Specific disposition, be saved in pipeline tables of data.

According to the three-dimensional tube line model in step S016, read the parameter of above-mentioned pipeline table configuration, extract by pipeline kind Guan Dian, pipeline are saved in different mdb form pipe point data table that (pipe point table naming rule: XXPOINT, such as feed pipe point respectively Table JSPOINT) and pipeline tables of data (pipeline table naming rule: XXLINE, such as water-supply line table: JSLINE) in, wherein pipeline Kind, according to country " pipeline element classification code and symbolic formulation " CH/T 1036-2015, is divided into 9 big class electric power (DL), electricity Letter (DX), feeds water (JS), draining (PS), combustion gas (RQ), heating power (RL), industry (GY), integrated pipe canal (ZH), other (QT).Pipe Line group is classified according to surveying district's actual conditions, and encodes using the acronym of classification as pipeline group.

S018, the pipeline information extracted according to step S017, reads the system configuration parameter pipe point to extracting, pipeline data Check, and will check that result is according to type of error list.Pipeline check process: 1. in data checks item configures, data The scope of examination includes the following:

2. according to above check item, check gauge typical value is set for the scope of examination.

A. structure inspection: according to each pipeline table structure of definition, to the field quantity in table, field name, type, length Degree, precision check, check whether the content of non-NULL field exists the situation of null attribute simultaneously.B. data uniqueness inspection: Mainly checking in pipe point pipeline table unique value field whether there is identical recordings, the error message checked is with the form of form Performance；C. range check, is configured the maximin of pipe point pipeline numerical value.D. pipeline connectivity checks: according to setting Connectivity checks table (record do not allow connection pipe point feature and adjunct, such as discharge outlet, water inlet) check submit to number According to whether there is the situation in the local appearance connection not allowing connection.E. requirement during uniqueness inspection arranges pipe point and pipeline table The unique field of data.F. fixterm input checking: mainly check and specify that the fill substance in field is at pipe point pipeline table No fill in by fixing input item content, fill in without by regulation, show at error, such as: draining Feature field only allows fill in (inspection shaft material: concrete, masonry, plastics etc.) if fill in not within, then report an error；G. dotted line pair Should check: go to check the starting point pipeline period field in line table and terminal pipeline according to numbering in the pipe point number field in a table Whether exist in period field, if there is no then pointing out, inspection database is wrong；H. line point correspondence proving: according in line table Whether starting point pipeline period field and terminal pipeline period field contents go in the pipe point number field in the table of checkpoint in numbering Exist, lack the some attribute record of certain physical prospecting period if there is no then prompting；I. draining flows to check: mainly check draining Water (flow) direction in class pipeline the most rationally (generally assumes that draining flows to less one from one end that pipeline shaft bottom absolute altitude is bigger End), the first maximum absolute altitude tolerance limit of input (i.e. flowing to the Maximum tolerance of terminal absolute altitude-flow to starting point absolute altitude), then according to pipeline In the shaft bottom absolute altitude difference of 2 judge line table " flows to " reasonability of field contents, check irrational information (record Entity ID, the information table at place, warning message describes) show with the form of form；J. pipeline overlength checks: specify pipeline to allow Maximum length, check whether the length of pipe section of pipeline exceedes the largest tube line length of setting, the overlength pipeline checked is with report The form performance of table；K. feature adjunct atural object structure building coding check: main " feature ", " attached checked in pipe point table Thing " whether field occurs in that the property value of paradox, such as: if filling in " reducing " in " feature ", and fill out at " adjunct " Write " valve well ", carry out coding check simultaneously, main check each tubing, the pipeline object coding in pipe point pipeline table whether with In pipeline surveying code consistent (the including that line coding, some coding, pipe point naming rule are the most consistent in tubing) of regulation；

3. according to the parameter traversals pipeline data set in above-mentioned inspection configuration item, the record in tables of data is examined Look into, and misregistration result.

4. according to checking that error message is revised by result one by one, check again for after having revised, until there is no error logging.

S019, the pipeline model rendering pipeline 2 d plane picture drawn according to step S016, extracts according to step S018 Pipeline information mark pipeline kind, material and caliber information, finally give pipeline result map；Read pipeline database and generate pipe Line generates two dimensional pipeline plane, according to becoming figure to require the information such as mark pipeline kind, material, caliber, generates conduit line map.

2 d plane picture generates process: 1. create into figure figure layer according to the pipeline kind in mdb database.Figure layer title with Tables of data table name is identical (such as: draining (PS) type, to be created line PSLINE, puts PSPOINT, pipe point annotation PSMark, pipeline annotation Tetra-figure layers of PSM).2. pipe dot pattern is according to the X in tables of data, and Y coordinate adds entity point, dot pattern symbol according to feature or In adjunct field, type adds the Symbol Style.3. pipeline figure is numbered according to the origin number in tables of data and terminal, respectively In reading point table, the point of reference numeral is as pipeline head and terminal rendering pipeline figure, and the pattern of pipeline, color are line classification Color defined in table.

S110, by pipeline result map, the graphics of pipeline model, pipeline 2 d plane picture preserves to database.According to Engine request submits to outputting result to include pipeline electronic plane figure, pipe isometric and pipeline database necessity achievement.

(2) pipeline data are carried out classification according to pipeline classification and store into GIS spatial data library format.

This step can be directly pipeline data to be stored as GIS spatial data form, it is also possible to is by above-mentioned steps The pipeline cad data set up is converted to GIS spatial data.

Wherein pipeline cad data is changed and is included step:

S021, the layering of pipeline cad data is converted to GIS spatial data, and records corresponding attribute item.

The annotation that can directly copy is copied in the text of GIS spatial data；For the note that can not directly copy Note, analyzes the intercharacter length of No. ID, size and two of annotation in text, obtains text message, will delete in text message Space character as the text attribute of annotation, text attribute is write in the text of GIS spatial data；

S022, the attribute information in pipeline cad data is assigned to corresponding GIS spatial data, as GIS spatial data Attribute information.

The process that pipeline data are directly stored as GIS spatial data form is:

S11, the relevant configuration table in reading field data storehouse, judge whether deposit in field data storehouse according to pipeline group coding In tables of data and the type of tables of data of corresponding pipeline,

S12, if there is no continuing to search for next pipeline type；

S13, if there is a table, then in a table, gradually read record, the X-coordinate and the Y coordinate that obtain every record are raw Become Point type；

S14, if there is line table, then according to starting point pipeline period and terminal pipeline period, at the pipe of corresponding pipeline type Point field data table is searched X-coordinate and the Y coordinate of corresponding record, generates line sections point and end of pipe respectively, according to pipeline Duan Qidian and end of pipe generate line sections PolyLine type；

All pipeline type in S15, traversal allocation list repeat step S12, and S13, S14 generate all of pipe point and pipeline Section spatial data table.

Concrete implementation process is as follows:

Field data acquisition pipeline data are the attribute data form of Microsoft Access form storage.As in figure 2 it is shown, pipe In point data table, (pipeline table is named for (pipe point table naming rule: XXPOINT, such as feed pipe point table JSPOINT) and pipeline tables of data Rule: XXLINE, such as water-supply line table: JSLINE) in, wherein pipeline kind is according to country " pipeline element classification code and symbol Number express " CH/T 1036-2015, it is divided into 9 big class electric power (DL), telecommunications (DX), feed water (JS), draining (PS), combustion gas (RQ), heating power (RL), industry (GY), integrated pipe canal (ZH), other (QT).Pipeline group is carried out point according to surveying district actual conditions Class, and encode using the acronym of classification as pipeline group.

Line classification such as table 1:

Table 1 line classification table

Point pipeline tables of data is for describing the association attributes of point pipeline, and (XX is that pipeline group is compiled to naming rule xxPoint Code).The structure of pipe point data table such as table 2:

Table 2 point pipeline gathers data community table

Line sections tables of data is for describing the association attributes of line sections, named xxLine.The structure of line sections tables of data Such as table 3:

Table 3 line sections gathers data community table

As it is shown on figure 3, according to above pipe point, line sections tables of data form, first pass through self-compiling program and generate pipe space Data form.Specifically comprise the following steps that and 1. read allocation list (being shown in Table 4), according to pipeline group coding judge field data storehouse is The no type (point or line) that there is tables of data and tables of data, 2. if there is a table, then obtains X-coordinate and Y coordinate generates Point type；3. if there is no continuing to search for next pipeline type；4. if there is line table, then according to according to starting point pipeline Period and terminal pipeline period, search X-coordinate and the Y coordinate of correspondence in XXPoint table respectively, generate rising of line sections respectively Point S_Point and end of pipe E_Point, generates line sections PolyLine type according to S_Point and E_Point；5. travel through All pipeline type in allocation list repeat the most 4. to generate all of pipe point and line sections spatial data table.

Table 4 allocation list

Pipeline data are stored into pipe space data form and (originally sentences Arcgis Personal Geodatabase lattice As a example by formula), wherein, spatial data table name form is with field data acquisition tables title identical (as shown in Figure 2).Space point pipeline Table field form such as table 5:

Table 5 space point pipeline table

Space line sections table field formula such as table 6:

Table 5 space pipeline segment table

Wherein, Geometry type is the graphical information of binary form storage, comprises georeferencing coordinate system and figure Information, is divided into point (Point), scope (Extent), circle (Circle), multi-thread (Polyline), face (Polygon) 5 type.

(3) the conduit line map graphic data of spatial data library format, it is converted into JSON ((JavaScript Object Notation)) data switched data form.

Data Format data volume is relatively big, transmits in data, transfer, and during storage, ratio is relatively time-consuming, being quickly total to data Enjoy the restriction effect that serves, need that this is write Conversion of Spatial Data and become the relatively small file that takes up room, improve transfer rate.

By self-compiling program, the data in spatial data table are converted into formatting data by the sequencing of record, each Tables of data is stored in a JSON file.

As shown in Figure 4, all tables of data in spatial database are 1. traveled through；2. judge data table types (Point or Polyline), lay equal stress on the entitled title identical with spatial data table according to attribute list type step JSON format module；3. open Template adds Spatial data types, reference frame at file the beginning part with the form (" key: value ") of key-value pair, often organizes key assignments To separating with ", " respectively；4. adding list of fields in field groups (Fields), each field includes title (name), type (fieldtype), three groups of key-value pairs of length (length), often group key-value pair is used respectively ", " separate, three groups of key assignments are placed on " { } " In, adjacent field is split with ", ", and all of field forms key-value pair with field value respectively, and all field values are with " [] " includes； 5. adding data record in data group (features), all records and data group (features) form key-value pair, all Record includes with " [] ", ", " segmentation between every record；Every record includes set of properties (attribute) and geometry (geometry) two subgroups, deposit the attribute list of record with the form of key-value pair in set of properties (attribute), each Between attribute (attribute: value) key-value pair with ", " separately, all of attribute key-value pair leaves in " { } "；Geometry (geometry) the JSON formatted data of geometry type is deposited in；6. traversal current spatial tables of data repeats the most 5. step, Preserving all of data record, finally the beginning and end at file adds " { ", " } " respectively, then preserves；7. at space number According to the most 6. storehouse processes all of spatial data table according to above step.

Wherein, point pipeline Geometry type conversion JSON form is as follows:

" x ": 95707.58069493785, " y ": 66290.38461433914,

" spatialReference ": { wkid:4821}}

Wherein " x " is X correspondence X-coordinate value, " y " corresponding Y-coordinate value.

Line sections Geometry type conversion JSON form is as follows:

" paths ": [[[-122.68,45.53], [-122.58,45.55]], " spatialReference ":

{wkid:4821}}

Wherein, the data group of " paths " corresponding point constituting line segment and space coordinates, the coordinate of each point uses [x, y] form.

(4) carry instrument by Hadoop to import data under hdfs data storage catalogue.

Carry-put order by Hadoop the JSON data of formatting to be imported under hdfs catalogue.Process in data Time, carry out data process with parallel working method, fast more a lot of than single process data processes.

Although the detailed description of the invention of the present invention is described by the above-mentioned accompanying drawing that combines, but not the present invention is protected model The restriction enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme, and those skilled in the art are not Need to pay various amendments or deformation that creative work can make still within protection scope of the present invention.

Claims (10)

1. a pipeline date storage method, is characterized in that, including step:

S1, pipeline data are carried out classification according to pipeline classification store into GIS spatial data form；

S2, the pipeline data of the GIS spatial data form described in step S1 are converted into JSON data form；

S3, utilize Hadoop the JSON data form described in step S2 is imported to data storage catalogue under.

A kind of pipeline date storage method the most according to claim 1, is characterized in that, described step S1 includes step:

S11, the relevant configuration table in reading field data storehouse, judge whether exist in field data storehouse according to pipeline group coding right Answer the tables of data of pipeline and the type of tables of data,

S12, if there is no continuing to search for next pipeline type；

S13, if there is a table, then in a table, gradually read record, obtain every record X-coordinate and Y coordinate generate Point type；

S14, if there is line table, then according to starting point pipeline period and terminal pipeline period, outside the pipe point of corresponding pipeline type Industry tables of data is searched X-coordinate and the Y coordinate of corresponding record, generates line sections point and end of pipe respectively, rise according to line sections Point and end of pipe generate line sections PolyLine type；

All pipeline type in S15, traversal allocation list repeat step S12, and S13, S14 generate all of pipe point and line sections is empty Between tables of data.

A kind of pipeline date storage method the most according to claim 1, is characterized in that, described step S1 includes step:

S01, collection pipeline data, set up pipeline CAD database；

S02, pipeline CAD database is converted to GIS spatial data form.

A kind of pipeline date storage method the most according to claim 3, is characterized in that, described step S01 includes step:

S011, utilizes the cloud data of laser scanner locality underground pipelines network；

S012, is forwarded to the cloud data obtained in step S011 in same coordinate system by Registration of Measuring Data；

S013, rejects the noise data in the cloud data under same coordinate system；

S014, takes out dilute process to the cloud data rejecting noise data in step S013；

S015, the cloud data after processing according to step S014 sets up pipeline surface model；

S016, the pipeline surface modeling rendering pipeline model generated according to step S015；

S017, the pipeline model drawn according to step S016, extracts pipeline information；

S018, the pipeline information extracted according to step S017, read the system configuration parameter pipe point to extracting, pipeline data are carried out Check, and will check that result is according to type of error list；

S019, the pipeline model rendering pipeline 2 d plane picture drawn according to step S016, the pipeline extracted according to step S018 Information labeling pipeline kind, material and caliber information, finally give pipeline result map；

S0110, by pipeline result map, the graphics of pipeline model, pipeline 2 d plane picture preserves to pipeline CAD database.

5. according to a kind of pipeline date storage method described in claim 2 or 4, it is characterized in that, described step S02 includes step Rapid:

S021, the layering of pipeline cad data is converted to GIS spatial data, and records corresponding attribute item；

S022, the attribute information in pipeline cad data is assigned to corresponding GIS spatial data, as the genus of GIS spatial data Property information.

A kind of pipeline date storage method the most according to claim 5, is characterized in that, described step S2 includes step:

S21, travel through all of spatial data table；

S22, judge data table types, generate JSON format module according to data table types and lay equal stress on entitled identical with spatial data table Title；

S23, open template at file the beginning part with the form of key-value pair: " key: value " adds Spatial data types, reference coordinate System, often group key-value pair separates with ", " respectively；

S24, in field groups add list of fields, each field includes title, type, three groups of key-value pairs of length, often organizes key assignments To separating with ", " respectively, three groups of key assignments are placed in " { } ", and adjacent field is split with ", "；All field values include with " [] ", All of field forms key-value pair with field value respectively；

S25, adding data record in data group, all records and data group form key-value pair, and all records include with " [] ", ", " segmentation between every record；Every record includes set of properties and two subgroups of geometry, with key-value pair in set of properties Form deposits the attribute list of record, and each attribute uses: attribute: value, key-value pair form, with ", " point between adjacent key-value pair Opening, all of attribute key-value pair leaves in " { } "；Geometry is deposited the JSON formatted data of geometry type；

S26, traversal current spatial tables of data repeat S24, S25 step, preserve all of data record, finally in the beginning of file Add " { ", " } " respectively with ending, then preserve；

S27, in spatial database according to step S22, S23, S24, S25, S26 process all of spatial data table.

A kind of pipeline date storage method the most according to claim 6, is characterized in that, described step S3 includes: step

The JSON data of formatting are imported under catalogue by the S31 ,-put order carried by Hadoop.

A kind of pipeline date storage method the most according to claim 7, is characterized in that, the Hadoop in described step S31 -the put carried orders the operation that works in a parallel fashion.

A kind of pipeline date storage method the most according to claim 8, is characterized in that, described step S011 includes step:

S111, draws piping lane trend on topographic map, region to be measured is divided into several grids；

S112, includes a survey station and at least three target point in each grid；

A113, to all grids in scope to be measured, sets up three-dimensional laser scanner one by one；

S114, in each grid, sets up three-dimensional cartesian coordinate system with laser scanner for initial point: wherein, and X-axis is in transversal scanning face In, Y-axis is vertical with X-axis in transversal scanning face, and Z axis is vertical with transversal scanning face；Utilize laser scanner measurement laser scanning A survey station in the grid of instrument place and the coordinate of three target points；

S115, repetition step S112 is to step S114, until all grids are the most measured complete.

A kind of pipeline date storage method the most according to claim 9, is characterized in that, the data in described step S012 Registration utilizes boolean's sand Seven-parameter to carry out Registration of Measuring Data；Described step S013 utilizes macroscopic examination method or curve inspection technique Or string high differentiation carries out noise rejecting；The dilute distance of taking out taking out dilute process in described step S014 is 5cm, described step S015 The middle method setting up the utilization of pipeline surface model is Di Luoni triangle terrain model.