CN117115063A - Multi-source data fusion application method - Google Patents
Multi-source data fusion application method Download PDFInfo
- Publication number
- CN117115063A CN117115063A CN202311271724.5A CN202311271724A CN117115063A CN 117115063 A CN117115063 A CN 117115063A CN 202311271724 A CN202311271724 A CN 202311271724A CN 117115063 A CN117115063 A CN 117115063A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- data
- cloud data
- dimensional model
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000011218 segmentation Effects 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000012986 modification Methods 0.000 claims abstract description 7
- 230000004048 modification Effects 0.000 claims abstract description 7
- 238000001914 filtration Methods 0.000 claims description 30
- 238000010586 diagram Methods 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 15
- 238000011176 pooling Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 10
- 230000008447 perception Effects 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 7
- 239000003086 colorant Substances 0.000 claims description 6
- 238000009877 rendering Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 239000012633 leachable Substances 0.000 claims description 4
- 238000004040 coloring Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 5
- 238000013461 design Methods 0.000 abstract description 4
- 230000010354 integration Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000002950 deficient Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- QERYCTSHXKAMIS-UHFFFAOYSA-M thiophene-2-carboxylate Chemical compound [O-]C(=O)C1=CC=CS1 QERYCTSHXKAMIS-UHFFFAOYSA-M 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Medical Informatics (AREA)
- Geometry (AREA)
- Databases & Information Systems (AREA)
- Biomedical Technology (AREA)
- Computer Graphics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Remote Sensing (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a multi-source data fusion application method, which comprises the following steps: step one, processing point cloud data to obtain target point cloud data; step two, processing the image to obtain a target image; step three, matching the target point cloud data with the target image to obtain modeling data; step four, model reconstruction: constructing a three-dimensional model and a two-dimensional model of the detected area based on the modeling data; fifthly, model modification: modifying the two-dimensional model and the three-dimensional model; step six, model display: and integrally displaying the two-dimensional space data of the two-dimensional model and the three-dimensional space data of the three-dimensional model. The application has simple structure and reasonable design, adopts a method of rough segmentation and fine adjustment to process point cloud data, adopts a learnable weight to fuse historical images and newly added images, then fuses multi-source point cloud data and multi-source images to obtain modeling data, respectively establishes a two-dimensional model and a three-dimensional model, and integrates two-dimensional space data and three-dimensional space display.
Description
Technical Field
The application belongs to the technical field of geographic information, and particularly relates to a multi-source data fusion application method.
Background
In modern society, the variety and amount of data collected by people is increasing, and such data is often from various sources, including sensors, social media, mobile devices, and the like. How to fuse these data together for analysis and application has become an important technical challenge.
Two-dimensional integration is an emerging technology, and can integrate data from different sources into a unified model, so that visualization and analysis can be better performed.
In the aspect of geographic information acquisition and display, multi-source data fusion is realized, geographic geological data is comprehensively utilized, a comprehensive application platform is formed, a geographic environment is intuitively analyzed, and the management of a data scene is realized, so that the method has important significance.
Disclosure of Invention
The application aims to solve the technical problems in the prior art, and provides a multi-source data fusion application method which is simple in structure and reasonable in design, and is used for fusing multi-source point cloud data and multi-source images to obtain modeling data, respectively establishing a two-dimensional model and a three-dimensional model, integrating two-dimensional space data and three-dimensional space display, and enabling the two-dimensional space data and the three-dimensional space data to be located in the same coordinate system.
In order to solve the technical problems, the application adopts the following technical scheme: a multi-source data fusion application method is characterized in that:
step one, acquiring cloud data of a target point: inputting the first point cloud data, the second point cloud data and the historical point cloud data into a rough segmentation network, and enhancing the description of pixel characteristics according to the relation between the learning pixels of the rough segmentation network and the characteristics of the object region to obtain a rough segmentation result; fusing the rough segmentation result and the historical point cloud data to obtain fusion point cloud, inputting the fusion point cloud into a fine tuning network, and outputting a precise segmentation result by the fine tuning network to obtain target point cloud data;
step two, obtaining a target image: respectively extracting features of the first newly added image, the second newly added image and the historical image to obtain a first feature map F 1 Second characteristic diagram F 2 And a third characteristic diagram F 3 For the first characteristic diagram F 1 And a second characteristic diagram F 2 Fusing to obtain a fused feature map F R The method comprises the steps of carrying out a first treatment on the surface of the Based on the multi-layer perception mlp model, a leachable weight is output, and based on the leachable weight, a third feature map F is fused 3 And fusion of feature map F R Obtaining a target image;
step three, matching the target point cloud data with the target image to obtain modeling data;
step four, model reconstruction: constructing a three-dimensional model and a two-dimensional model of the detected area based on the modeling data;
fifthly, model modification: modifying the two-dimensional model and the three-dimensional model;
step six, model display: and integrally displaying the two-dimensional space data of the two-dimensional model and the three-dimensional space data of the three-dimensional model.
The application method for multi-source data fusion is characterized by comprising the following steps: the specific method of the first step is as follows:
step 101, first aviation equipment collects first newly-increased point cloud data of a detected area, preprocesses the first newly-increased point cloud data, corrects the first newly-increased point cloud data, and obtains first point cloud data;
102, acquiring second newly-increased point cloud data of a detected area by first vehicle-mounted equipment, preprocessing the second newly-increased point cloud data, and correcting the second newly-increased point cloud data to obtain second point cloud data;
step 103, acquiring historical point cloud data of a detected area;
104, the first point cloud data, the second point cloud data and the historical point cloud data form a point cloud data set, the point cloud data set is divided into a training set, a verification set and a test set, and corresponding labels are added to the point cloud data of the training set;
step 105, constructing a segmentation network, inputting point cloud data of a training set into the segmentation network to obtain a prediction segmentation result, and adjusting network parameters of the segmentation network according to the prediction segmentation result until a training stop condition is met to obtain a trained segmentation network;
step 106, respectively inputting the first point cloud data and the second point cloud data into a segmentation network to obtain a rough segmentation result, fusing the rough segmentation result with the historical point cloud data to obtain fused point cloud, and slicing the fused point cloud;
step 107, building a fine tuning network: and inputting the slice data into a fine tuning network, and outputting a precise segmentation result by the fine tuning network to obtain target point cloud data.
The application method for multi-source data fusion is characterized by comprising the following steps: the specific method of the second step is as follows:
step 201, a second aviation device collects a first newly-added image of a detected area, and performs space three encryption on the preprocessed first newly-added image to obtain a first newly-added image;
step 202, a second vehicle-mounted device collects a second newly-added image of a detected area, and performs space three encryption on the preprocessed second newly-added image to obtain a second newly-added image;
step 203, acquiring a history image of the detected area;
step 204, extracting features of the first newly added image and the second newly added image to obtain a first feature map F 1 And a second characteristic diagram F 2 Extracting features of the historical images to obtain a third feature map F 3 ；
Step 205, using BCA network to perform a first feature map F 1 And a second characteristic diagram F 2 Fusing to obtain a fused feature map F R ；
206, constructing a multi-layer perception mlp model;
step 207, for the fusion feature map F R Global maximum pooling processing is carried out to obtain a first pooled value, and a first channel dimension vector C is constructed by using the first pooled value FR ；
Step 208, for the third feature map F 3 Performing global average pooling processing to obtain a second pooled value, and constructing a second channel dimension vector C by using the second pooled value F3 ；
Step 209, the first channel dimension vector C FR And a second channel dimension vector C F3 Input multi-layer perceptual mlp model, mlp model output weight ω 1 ；
Step 2010, based on formula f=ω 1 ·F R +(1-ω 1 )·F 3 For fusion of feature map F R And a third characteristic diagram F 3 And carrying out weighted fusion to obtain the target image.
The application method for multi-source data fusion is characterized by comprising the following steps: the specific method for matching the target point cloud data with the target image comprises the following steps:
step 301, determining a matching domain: the corner of the mth target image is read by adopting a CornerNet model, a target frame is formed by the corner, and the ground coordinate X of each corner is obtained by conversion j The ground coordinate connecting lines of all the corner points form a matching domain;
step 302, determining a match line: dividing one side n of the target frame equally to obtain n+1 parallel lines, randomly selecting a point on each parallel line, and converting to obtain the ground coordinate X of each point d Ground coordinates X of each point d Connected to form a first match line;
step 303, finding a target point Yun Ziji corresponding to the matching domain in the target point cloud, and thenAt least 3 coordinates X with the ground are determined in the cloud subset d Corresponding coordinate points form a second matching line, and a rotation angle theta between the first matching line and the second matching line is calculated;
step 304, the target point cloud subset is corresponding to the rotation angle θ, and the rotated target point Yun Ziji is added into the feature point set of the mth target image to obtain a complete feature point set of the mth target image;
step 305, repeating steps 301-304 to complete the matching of all target point cloud data and target images.
The application method for multi-source data fusion is characterized by comprising the following steps: the modifying two-dimensional model in the fifth step comprises the following steps: detecting an invalid value region of the two-dimensional model and filling; repairing a noise area of the two-dimensional model; removing suspended matters of the two-dimensional model; setting a layering number, respectively corresponding a plurality of colors to a plurality of layering data, and performing layering coloring rendering on the two-dimensional model.
The application method for multi-source data fusion is characterized by comprising the following steps: the modification three-dimensional model in the fifth step comprises the following steps: detecting an invalid value region of the three-dimensional model and filling; repairing a noise area of the three-dimensional model; removing suspended matters of the three-dimensional model; setting a layering number, respectively corresponding a plurality of colors to a plurality of layering data, and rendering the layering color on the three-dimensional model; and (5) carrying out automatic dodging and color homogenizing treatment on the three-dimensional model.
The application method for multi-source data fusion is characterized by comprising the following steps: the segmentation network in the first step adopts a V-net network, and a pooling layer, a 1 multiplied by 1 convolution layer and 3 multiplied by 3 convolution layers with different expansion rates are sequentially arranged at the tail of an encoder of the V-net network.
The application method for multi-source data fusion is characterized by comprising the following steps: the fine tuning network in step 106 employs a U-net network that employs a weighted sum of the boundary loss function and the two-class cross entropy loss as the overall loss function.
The application method for multi-source data fusion is characterized by comprising the following steps: the two-dimensional space data comprises vector data, elevation data, image data and inclination data; the three-dimensional space data includes model data and tilt data.
The application method for multi-source data fusion is characterized by comprising the following steps: the first step of preprocessing the point cloud data comprises full-automatic point cloud filtering, wherein the full-automatic point cloud filtering comprises self-adaptive filtering, leveling filtering, smoothing filtering, fusion filtering, general filtering, elevation filtering and profile filtering.
Compared with the prior art, the application has the following advantages:
1. the application has simple structure, reasonable design and convenient realization, use and operation.
2. When the point cloud data is processed, the point cloud data is segmented by adopting the 3D convolution segmentation network in the first stage, the segmentation result is fused with the historical point cloud data in the channel dimension and is input into the fine tuning network, and then the 2D convolution fine tuning network is used for fine tuning the segmentation result, so that the network can fully utilize the characteristics of the three-dimensional convolution and the two-dimensional convolution, and the segmentation precision is improved.
3. The application adopts the BCA network to perform feature fusion on the first newly added image and the second newly added image to obtain the fusion feature map, updates the network weight based on the pooling values of the fusion feature map and the historical image, further updates the whole multi-layer perception mlp model, finally obtains the target image with the same size as the input image, and has good use effect.
4. In the application, the two-dimensional space data and the three-dimensional space data are integrally displayed.
In summary, the application has simple structure and reasonable design, adopts a method of rough segmentation and fine adjustment to process point cloud data, adopts a learnable weight to fuse historical images and newly added images, fuses multi-source point cloud data and multi-source images to obtain modeling data, respectively establishes a two-dimensional model and a three-dimensional model, and integrates two-dimensional space data and three-dimensional space display.
The technical scheme of the application is further described in detail through the drawings and the embodiments.
Drawings
Fig. 1 is a flowchart of a method for acquiring cloud data of a target point according to the present application.
Fig. 2 is a flowchart of a method for acquiring a target image according to the present application.
Fig. 3 is a flow chart of the method of the present application.
Detailed Description
The present application will be described in further detail with reference to the accompanying drawings.
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present application. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the application described herein may be capable of being practiced otherwise than as specifically illustrated and described. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Spatially relative terms, such as "above … …," "above … …," "upper surface at … …," "above," and the like, may be used herein for ease of description to describe one device or feature's spatial location relative to another device or feature as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "above" or "over" other devices or structures would then be oriented "below" or "beneath" the other devices or structures. Thus, the exemplary term "above … …" may include both orientations of "above … …" and "below … …". The device may also be positioned in other different ways (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
As shown in fig. 1-3, the present embodiment includes a multi-source data fusion application method, which is characterized in that:
step one, acquiring cloud data of a target point: inputting the first point cloud data, the second point cloud data and the historical point cloud data into a rough segmentation network, and enhancing the description of pixel characteristics according to the relation between the learning pixels of the rough segmentation network and the characteristics of the object region to obtain a rough segmentation result; and fusing the rough segmentation result and the historical point cloud data to obtain fusion point cloud, inputting the fusion point cloud into a fine tuning network, and outputting a precise segmentation result by the fine tuning network to obtain target point cloud data.
In one possible embodiment, the specific method for acquiring the cloud data of the target point is as follows:
step 101, first aviation equipment collects first newly-increased point cloud data of a detected area, preprocesses the first newly-increased point cloud data, corrects the first newly-increased point cloud data, and obtains first point cloud data;
102, acquiring second newly-increased point cloud data of a detected area by first vehicle-mounted equipment, preprocessing the second newly-increased point cloud data, and correcting the second newly-increased point cloud data to obtain second point cloud data;
the preprocessing of the point cloud data comprises full-automatic point cloud filtering, wherein the full-automatic point cloud filtering comprises self-adaptive filtering, leveling filtering, smoothing filtering, fusion filtering, general filtering, elevation filtering reduction and profile filtering.
Step 103, acquiring historical point cloud data of a detected area;
104, the first point cloud data, the second point cloud data and the historical point cloud data form a point cloud data set, the point cloud data set is divided into a training set, a verification set and a test set, and corresponding labels are added to the point cloud data of the training set.
In actual use, the method for preprocessing the first newly added point cloud data and the second newly added point cloud data comprises normalization and denoising by using Gaussian filtering.
Normalization is to scale the point cloud data to the same scale range, subtracting the mean and dividing by the standard deviation. Denoising using gaussian filtering is denoising of point cloud data using gaussian filtering.
According to 8:1: the data set is divided into a training set, a verification set and a test set according to the proportion of 1.
It should be noted that, the labels corresponding to the point cloud data are used for characterizing the category of each coordinate point therein. Examples: the corresponding labels can be set to be three categories of the target A, the target B, the target C and the background to be identified, or two categories of the target and the background.
Taking building identification as an example, each coordinate point in the point cloud data may be labeled with a building and a background, the building is identified, and then each coordinate point is labeled as a building or a background.
In one possible embodiment, taking a data set composed of point cloud data obtained at one time as an example, the data set is divided into a total of 800 images. Of these, 640 were used for model training, 80 were used for test verification, and 80 were used for model testing.
In one possible embodiment, the point cloud data may be acquired by a laser radar, the point cloud data acquired by the laser radar is transmitted to a remote controller on the ground in real time, and then the remote controller on the ground is transmitted to a terminal device such as a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (personal digital assistant, PDA) and the like in real time, so that the acquisition and the real-time transmission of the point cloud data are realized, and the terminal device performs the point cloud data segmentation. The acquisition mode of the point cloud data in the embodiment of the application is not limited.
Step 105, constructing a segmentation network, inputting point cloud data of a training set into the segmentation network to obtain a prediction segmentation result, and adjusting network parameters of the segmentation network according to the prediction segmentation result until a training stop condition is met to obtain a trained segmentation network;
step 106, respectively inputting the first point cloud data and the second point cloud data into a segmentation network to obtain a rough segmentation result, fusing the rough segmentation result with the historical point cloud data to obtain fused point cloud, and slicing the fused point cloud;
step 107, building a fine tuning network: and inputting the slice data into a fine tuning network, and outputting a precise segmentation result by the fine tuning network to obtain target point cloud data.
In the first stage, the point cloud data is segmented by adopting a 3D convolution segmentation network, the segmentation result is fused with the history point cloud data in the channel dimension and is input into a fine tuning network, and then the segmentation result is fine tuned by using a 2D convolution fine tuning network, so that the network can fully utilize the characteristics of the three-dimensional convolution and the two-dimensional convolution, and the segmentation precision is improved.
The historical point cloud data is added in the training set, and the local geometrical feature distribution information can be fully expressed, so that when the segmentation network performs semantic segmentation on the first point cloud data and the second point cloud data, the method has good adaptability, and the segmentation accuracy is improved. The key points of the point cloud data are extracted by using the segmentation network, so that the representativeness of the point cloud data is ensured while the number of the point clouds is reduced, and the matching and fusion efficiency of the step three can be improved.
And fusing the rough segmentation result and the historical point cloud data, and fully utilizing abundant features of the building widely existing in the historical point cloud data when the point cloud of the rough segmentation result is sparse, so as to realize effective positioning of the building.
And using a fine tuning network to subdivide pixels, and further optimizing the segmentation result of the segmentation network.
In this embodiment, the partitioning network in the first step adopts a V-net network, and the end of the encoder of the V-net network is sequentially provided with a pooling layer, a 1×1 convolution layer and 3×3 convolution layers with different expansion rates.
In this embodiment, the fine tuning network in the first step uses a U-net network, and the U-net network uses a weighted sum of the boundary loss function and the two kinds of cross entropy loss as the overall loss function.
The U-net network refers to a convolutional network for biomedical image segmentation. V-net networks refer to fully-convolutional neural networks for three-dimensional medical image segmentation.
Step two, processing the image to obtain a target image;
step 201, a second aviation device collects a first newly-added image of a detected area, and performs space three encryption on the preprocessed first newly-added image to obtain a first newly-added image;
step 202, a second vehicle-mounted device collects a second newly-added image of a detected area, and performs space three encryption on the preprocessed second newly-added image to obtain a second newly-added image;
step 203, acquiring a history image of the detected area;
step 204, extracting features of the first newly added image and the second newly added image to obtain a first feature map F 1 And a second characteristic diagram F 2 Extracting features of the historical images to obtain a third feature map F 3 。
The feature extraction module has a deep neural network built therein, and its feature extraction method is well known to those skilled in the art, so its specific implementation is not described too much.
Step 205, using BCA network to perform a first feature map F 1 And a second characteristic diagram F 2 Fusion is performed. The BCA network adopts a convolution layer to fuse the first characteristic diagram F 1 And a second characteristic diagram F 2 The BCA network uses a cross entropy loss function to supervise and an attention loss function to enhance the cooperativity between edge detection and semantic segmentation. The weights of the cross entropy loss function and the attention loss function are 1 and 0.5, respectively.
The English language of BCA is fully called: boundary guided Context Aggregation module, the BCA translates into a border guide network.
206, constructing a multi-layer perception mlp model;
the English of the MLP is called Multilayer Perceptron, the MLP is translated into a multi-layer perceptron, and the MLP is a feedforward artificial neural network model.
Step 207, for the fusion feature map F R Global maximum pooling processing is carried out to obtain a first pooled value, and a first channel dimension vector C is constructed by using the first pooled value FR ；
Step 208, for the third feature map F 3 Performing global average pooling processing to obtain a second pooled value, and constructing a second channel dimension vector C by using the second pooled value F3 ；
Step 209, the first channel dimension vector C FR And a second channel dimension vector C F3 Input multi-layer perceptual mlp model, mlp model output weight ω 1 ；
Step 2010, based on formula f=ω 1 ·F R +(1-ω 1 )·F 3 For fusion of feature map F R And a third characteristic diagram F 3 And carrying out weighted fusion to obtain the target image.
First for the fusion characteristic diagram F with the same channel number R And a third characteristic diagram F 3 Respectively carrying out pooling operation, constructing a channel dimension vector by using the obtained pooling value, and carrying out first channel dimension vector C FR And a second channel dimension vector C F3 Input multi-layer perceptual mlp model, mlp model output weight ω 1 。
Obtaining a learnable weight omega based on the pooling values 1 Therefore, if the image features are different, the pooling values are different, and the corresponding weight omega can be learned 1 Differently, the learnable weights ω are thus updated based on the multi-layer perceptions mlp model 1 Finally, the target image with the same size as the input image is obtained, and the use effect is good.
The first newly-added image acquired by the second aviation equipment is rich in information of the top area of the first newly-added image, but the side edge of the first newly-added image is deficient in information even in holes due to shooting angles or shielding. The second newly-increased image that second vehicle equipment gathered is rather than it, and the information in second newly-increased image top region is deficient even hole, but side information is complete, and the information is abundant. Therefore, the first newly-added image and the second newly-added image are fused, geometric constraint information with more abundant details can be provided, meanwhile, the historical images are fused, when the characteristic information of the first newly-added image and the second newly-added image is sparse, the abundant characteristics of the building widely existing in the historical images are fully utilized, the blank of the image information is avoided, and the effective positioning of the building is realized.
Step three, matching the target point cloud data with the target image to obtain modeling data, wherein the specific method comprises the following steps:
step 301, determining a matching domain: the corner of the mth target image is read by adopting a CornerNet model, a target frame is formed by the corner, and the ground coordinate X of each corner is obtained by conversion j The ground coordinate connecting lines of the corner points form a matching domain.
It should be noted that the CornerNet model predicts two sets of thermodynamic diagrams through the convolutional network, representing the top left and bottom right positions of the target image, respectively. The left upper corner position and the right lower corner position are corner points, and form a target frame.
Step 302, determining a match line: dividing one side n of the target frame equally to obtain n+1 parallel lines, randomly selecting a point on each parallel line, and converting to obtain the ground coordinate X of each point d Ground coordinates X of each point d Connected to form a first match line;
step 303, finding target points Yun Ziji corresponding to the matching domain in the target point cloud, and determining at least 3 coordinates X with the ground in the target point cloud subset d Corresponding coordinate points form a second matching line, and a rotation angle theta between the first matching line and the second matching line is calculated;
step 304, the target point cloud subset is corresponding to the rotation angle θ, and the rotated target point Yun Ziji is added into the feature point set of the mth target image to obtain a complete feature point set of the mth target image;
step 305, repeating steps 301-304 to complete the matching of all target point cloud data and target images.
And rotating the target point cloud data corresponding to the building to enable the target point cloud data to be matched and fused with the target image corresponding to the building, so that fusion of the air-ground image is completed, and the centimeter-level precision information brought by the target point cloud data is filled under the building frame provided by the target image, so that the positioning precision of the building can be improved.
Step four, model reconstruction: constructing a three-dimensional model and a two-dimensional model of the detected area based on the modeling data; it should be noted that, the model reconstruction is to build a three-dimensional model and a two-dimensional model of the building in the area to be tested based on the two-dimensional integration technology.
Fifthly, model modification: modifying the two-dimensional model and the three-dimensional model;
modifying the two-dimensional model includes: detecting an invalid value region of the two-dimensional model and filling; repairing a noise area of the two-dimensional model; removing suspended matters of the two-dimensional model; setting a layering number, respectively corresponding a plurality of colors to a plurality of layering data, and performing layering coloring rendering on the two-dimensional model.
Modifying the three-dimensional model includes: detecting an invalid value region of the three-dimensional model and filling; repairing a noise area of the three-dimensional model; removing suspended matters of the three-dimensional model; setting a layering number, respectively corresponding a plurality of colors to a plurality of layering data, and rendering the layering color on the three-dimensional model; and (5) carrying out automatic dodging and color homogenizing treatment on the three-dimensional model.
Step six, model display: and integrally displaying the two-dimensional space data of the two-dimensional model and the three-dimensional space data of the three-dimensional model based on a two-dimensional and three-dimensional integration technology.
The two-dimensional space data comprises vector data, elevation data, image data and inclination data; the three-dimensional space data includes model data and tilt data.
The two-dimensional integrated technology is a new generation GIS technology, and English of GIS is fully called: geographic Information System to a geographic information system. In short, the two-dimensional and three-dimensional integrated technology can integrate two-dimensional space data and three-dimensional space data in the GIS on the same platform. The built two-dimensional data can be directly used on the three-dimensional platform, and the two-dimensional space data can be directly visualized in the three-dimensional scene without any conversion processing. Simultaneously, in the process of using the two-dimensional and three-dimensional integrated technology, a user can directly operate the data of the building on the two-dimensional GIS map, and simultaneously, the three-dimensional GIS map of the building can be synchronously generated in the three-dimensional scene.
For example: when a user builds position data of a building on a two-dimensional GIS map, a topological relation of the building in a two-dimensional scene can be established through a two-dimensional and three-dimensional integrated technology, and three-dimensional GIS maps of the building can be synchronously generated by acquiring elevation data of the building in the two-dimensional scene.
Wherein the contents not described in detail in the specification belong to the prior art known to those skilled in the art.
The foregoing is merely an embodiment of the present application, and the present application is not limited thereto, and any simple modification, variation and equivalent structural changes made to the foregoing embodiment according to the technical matter of the present application still fall within the scope of the technical solution of the present application.
Claims (10)
1. A multi-source data fusion application method is characterized in that:
step one, acquiring cloud data of a target point: inputting the first point cloud data, the second point cloud data and the historical point cloud data into a rough segmentation network, and enhancing the description of pixel characteristics according to the relation between the learning pixels of the rough segmentation network and the characteristics of the object region to obtain a rough segmentation result; fusing the rough segmentation result and the historical point cloud data to obtain fusion point cloud, inputting the fusion point cloud into a fine tuning network, and outputting a precise segmentation result by the fine tuning network to obtain target point cloud data;
step two, obtaining a target image: respectively extracting features of the first newly added image, the second newly added image and the historical image to obtain a first feature map F 1 Second characteristic diagram F 2 And a third characteristic diagram F 3 For the first characteristic diagram F 1 And a second characteristic diagram F 2 Fusing to obtain a fused feature map F R The method comprises the steps of carrying out a first treatment on the surface of the Based on the multi-layer perception mlp model, a leachable weight is output, and based on the leachable weight, a third feature map F is fused 3 And fusion of feature map F R Obtaining a target image;
step three, matching the target point cloud data with the target image to obtain modeling data;
step four, model reconstruction: constructing a three-dimensional model and a two-dimensional model of the detected area based on the modeling data;
fifthly, model modification: modifying the two-dimensional model and the three-dimensional model;
step six, model display: and integrally displaying the two-dimensional space data of the two-dimensional model and the three-dimensional space data of the three-dimensional model.
2. A multi-source data fusion application method as defined in claim 1, wherein: the specific method of the first step is as follows:
step 101, first aviation equipment collects first newly-increased point cloud data of a detected area, preprocesses the first newly-increased point cloud data, corrects the first newly-increased point cloud data, and obtains first point cloud data;
102, acquiring second newly-increased point cloud data of a detected area by first vehicle-mounted equipment, preprocessing the second newly-increased point cloud data, and correcting the second newly-increased point cloud data to obtain second point cloud data;
step 103, acquiring historical point cloud data of a detected area;
104, the first point cloud data, the second point cloud data and the historical point cloud data form a point cloud data set, the point cloud data set is divided into a training set, a verification set and a test set, and corresponding labels are added to the point cloud data of the training set;
step 105, constructing a segmentation network, inputting point cloud data of a training set into the segmentation network to obtain a prediction segmentation result, and adjusting network parameters of the segmentation network according to the prediction segmentation result until a training stop condition is met to obtain a trained segmentation network;
step 106, respectively inputting the first point cloud data and the second point cloud data into a segmentation network to obtain a rough segmentation result, fusing the rough segmentation result with the historical point cloud data to obtain fused point cloud, and slicing the fused point cloud;
and 107, constructing a fine tuning network, inputting the slice data into the fine tuning network, and outputting a precise segmentation result by the fine tuning network to obtain target point cloud data.
3. A multi-source data fusion application method as defined in claim 1, wherein: the specific method of the second step is as follows:
step 201, a second aviation device collects a first newly-added image of a detected area, and performs space three encryption on the preprocessed first newly-added image to obtain a first newly-added image;
step 202, a second vehicle-mounted device collects a second newly-added image of a detected area, and performs space three encryption on the preprocessed second newly-added image to obtain a second newly-added image;
step 203, acquiring a history image of the detected area;
step 204, extracting features of the first newly added image and the second newly added image to obtain a first feature map F 1 And a second characteristic diagram F 2 Extracting features of the historical images to obtain a third feature map F 3 ；
Step 205, using BCA network to perform a first feature map F 1 And a second characteristic diagram F 2 Fusing to obtain a fused feature map F R ；
206, constructing a multi-layer perception mlp model;
step 207, for the fusion feature map F R Global maximum pooling processing is carried out to obtain a first pooled value, and a first channel dimension vector C is constructed by using the first pooled value FR ；
Step 208, for the third feature map F 3 Performing global average pooling processing to obtain a second pooled value, and constructing a second channel dimension vector C by using the second pooled value F3 ；
Step 209, the first channel dimension vector C FR And a second channel dimension vector C F3 Input multi-layer perceptual mlp model, mlp model output weight ω 1 ；
Step 2010, based on formula f=ω 1 ·F R +(1-ω 1 )·F 3 For fusion of feature map F R And a third characteristic diagram F 3 And carrying out weighted fusion to obtain a target image F.
4. A multi-source data fusion application method as defined in claim 1, wherein: the specific method for matching the target point cloud data with the target image comprises the following steps:
step 301, determining a matching domain: the corner of the mth target image is read by adopting a CornerNet model, a target frame is formed by the corner, and the ground coordinate X of each corner is obtained by conversion j The ground coordinate connecting lines of all the corner points form a matching domain;
step 302, determining a match line: dividing one side n of the target frame equally to obtain n+1 parallel lines, randomly selecting a point on each parallel line, and converting to obtain the ground coordinate X of each point d Ground coordinates X of each point d Connected to form a first match line;
step 303, finding target points Yun Ziji corresponding to the matching domain in the target point cloud, and determining at least 3 coordinates X with the ground in the target point cloud subset d Corresponding coordinate points form a second matching line, and a rotation angle theta between the first matching line and the second matching line is calculated;
step 304, the target point cloud subset is corresponding to the rotation angle θ, and the rotated target point Yun Ziji is added into the feature point set of the mth target image to obtain a complete feature point set of the mth target image;
step 305, repeating steps 301-304 to complete the matching of all target point cloud data and target images.
5. A multi-source data fusion application method as defined in claim 1, wherein: the modifying two-dimensional model in the fifth step comprises the following steps: detecting an invalid value region of the two-dimensional model and filling; repairing a noise area of the two-dimensional model; removing suspended matters of the two-dimensional model; setting a layering number, respectively corresponding a plurality of colors to a plurality of layering data, and performing layering coloring rendering on the two-dimensional model.
6. A multi-source data fusion application method as defined in claim 1, wherein: the modification three-dimensional model in the fifth step comprises the following steps: detecting an invalid value region of the three-dimensional model and filling; repairing a noise area of the three-dimensional model; removing suspended matters of the three-dimensional model; setting a layering number, respectively corresponding a plurality of colors to a plurality of layering data, and rendering the layering color on the three-dimensional model; and (5) carrying out automatic dodging and color homogenizing treatment on the three-dimensional model.
7. A multi-source data fusion application method according to claim 2, characterized in that: the segmentation network in the first step adopts a V-net network, and a pooling layer, a 1 multiplied by 1 convolution layer and 3 multiplied by 3 convolution layers with different expansion rates are sequentially arranged at the tail of an encoder of the V-net network.
8. A multi-source data fusion application method according to claim 2, characterized in that: the fine tuning network in the first step adopts a U-net network, and the U-net network adopts a weighted sum of a boundary loss function and two kinds of cross entropy loss as an integral loss function.
9. A multi-source data fusion application method according to claim 1 or 6, characterized in that: the two-dimensional space data comprises vector data, elevation data, image data and inclination data; the three-dimensional space data includes model data and tilt data.
10. A multi-source data fusion application method according to claim 1 or 6, characterized in that: the first step of preprocessing the point cloud data comprises full-automatic point cloud filtering, wherein the full-automatic point cloud filtering comprises self-adaptive filtering, leveling filtering, smoothing filtering, fusion filtering, general filtering, elevation filtering and profile filtering.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311271724.5A CN117115063A (en) | 2023-09-28 | 2023-09-28 | Multi-source data fusion application method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311271724.5A CN117115063A (en) | 2023-09-28 | 2023-09-28 | Multi-source data fusion application method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117115063A true CN117115063A (en) | 2023-11-24 |
Family
ID=88811108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311271724.5A Pending CN117115063A (en) | 2023-09-28 | 2023-09-28 | Multi-source data fusion application method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117115063A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117889781A (en) * | 2024-03-13 | 2024-04-16 | 深圳市高松科技有限公司 | EDM electrode rapid detection device |
| CN118097656A (en) * | 2024-04-01 | 2024-05-28 | 中创智元信息技术有限公司 | Spatial data construction method based on live-action three-dimension |
-
2023
- 2023-09-28 CN CN202311271724.5A patent/CN117115063A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117889781A (en) * | 2024-03-13 | 2024-04-16 | 深圳市高松科技有限公司 | EDM electrode rapid detection device |
| CN118097656A (en) * | 2024-04-01 | 2024-05-28 | 中创智元信息技术有限公司 | Spatial data construction method based on live-action three-dimension |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2021250975B2 (en) | Statistical point pattern matching technique | |
| KR102126724B1 (en) | Method and apparatus for restoring point cloud data | |
| Ham et al. | Visual monitoring of civil infrastructure systems via camera-equipped Unmanned Aerial Vehicles (UAVs): a review of related works | |
| Chen et al. | A methodology for automated segmentation and reconstruction of urban 3-D buildings from ALS point clouds | |
| US10127670B2 (en) | Computer vision systems and methods for detecting and modeling features of structures in images | |
| US20210027532A1 (en) | Primitive-based 3d building modeling, sensor simulation, and estimation | |
| CN113139453B (en) | Orthoimage high-rise building base vector extraction method based on deep learning | |
| CN117115063A (en) | Multi-source data fusion application method | |
| US11636649B2 (en) | Geospatial modeling system providing 3D geospatial model update based upon predictively registered image and related methods | |
| US20220091259A1 (en) | System using a priori terrain height data for interferometric synthetic aperture radar (ifsar) phase disambiguation and related methods | |
| CN114758337A (en) | Semantic instance reconstruction method, device, equipment and medium | |
| CN114219855A (en) | Point cloud normal vector estimation method and device, computer equipment and storage medium | |
| CN113920254B (en) | Monocular RGB (Red Green blue) -based indoor three-dimensional reconstruction method and system thereof | |
| CN113284144A (en) | Tunnel detection method and device based on unmanned aerial vehicle | |
| CN116030208A (en) | Method and system for building scene of virtual simulation power transmission line of real unmanned aerial vehicle | |
| Krooks et al. | WebGL visualisation of 3D environmental models based on finnish open geospatial data sets | |
| CN116403275B (en) | Method and system for detecting personnel advancing posture in closed space based on multi-vision | |
| CN116051980B (en) | Building identification method, system, electronic equipment and medium based on oblique photography | |
| CN110363863A (en) | A kind of the input data generation method and system of neural network | |
| CN113538579B (en) | Mobile robot positioning method based on unmanned aerial vehicle map and ground binocular information | |
| US11816793B2 (en) | Geospatial modeling system providing 3D geospatial model update based upon iterative predictive image registration and related methods | |
| CN115601517A (en) | Rock mass structural plane information acquisition method and device, electronic equipment and storage medium | |
| Widyaningrum et al. | Tailored features for semantic segmentation with a DGCNN using free training samples of a colored airborne point cloud | |
| Bai et al. | Application of unmanned aerial vehicle multi-vision image 3D modeling in geological disasters | |
| CN110910496B (en) | VR natural environment automatic construction method based on big data and AI |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |