WO2022237368A1 - 点云模型处理方法、装置及可读存储介质 - Google Patents

点云模型处理方法、装置及可读存储介质 Download PDF

Info

Publication number
WO2022237368A1
WO2022237368A1 PCT/CN2022/084109 CN2022084109W WO2022237368A1 WO 2022237368 A1 WO2022237368 A1 WO 2022237368A1 CN 2022084109 W CN2022084109 W CN 2022084109W WO 2022237368 A1 WO2022237368 A1 WO 2022237368A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
point cloud
target
cloud model
matching relationship
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/084109
Other languages
English (en)
French (fr)
Inventor
孙曦
宋振波
张永杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to US18/283,705 priority Critical patent/US12511826B2/en
Publication of WO2022237368A1 publication Critical patent/WO2022237368A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/56—Particle system, point based geometry or rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts

Definitions

  • the present disclosure relates to the technical field of image processing, and in particular to a point cloud model processing method, device and readable storage medium.
  • the multi-cluster 3D point cloud model is obtained by 3D reconstruction by taking multiple frames of images or video sequences of the object, and then the reconstructed multi-cluster 3D point cloud models are merged, and restored according to the merged 3D point cloud model The three-dimensional structure of the object.
  • the iterative nearest neighbor algorithm (Iterative Closest Point, ICP) is used to obtain the merging parameters, and the multi-cluster 3D point cloud models are merged according to the merging parameters.
  • ICP Intelligent Closest Point
  • the present disclosure provides a point cloud model processing method, device and readable storage medium.
  • an embodiment of the present disclosure provides a method for processing a point cloud model, including:
  • the target image pair and the first neighborhood image and the second neighborhood image corresponding to the target image pair are obtained; wherein the first image set includes A plurality of first images with a first visual effect; the second image set includes a plurality of second images with a second visual effect; the target image pair includes a target first image and a target first image whose similarity satisfies a first condition Two images; the first neighborhood image includes an image whose similarity with the target first image in the first image set satisfies a second condition; the second neighborhood image includes an image in the second image set that matches the target first image An image whose similarity to the target second image satisfies the third condition;
  • the first The point cloud model includes a three-dimensional point cloud model reconstructed in advance according to the first image set;
  • the second point cloud model includes a three-dimensional point cloud model reconstructed in advance according to the second image set;
  • the acquiring the target image pair and the first neighborhood image and the second neighborhood image corresponding to the target image pair according to the first image set and the second image set taken for the target scene include:
  • the acquisition of Merge parameters including:
  • the second matching relationship and the camera position corresponding to the first neighborhood image perform feature triangulation processing to obtain a third point cloud model; wherein the second matching relationship includes the first image of the target and the first neighborhood image A pixel point matching relationship between the two neighborhood images, the coordinate system of the third point cloud model is the same as that of the first point cloud model and the scale is the same;
  • the third matching relationship and the camera position corresponding to the second neighborhood image perform feature triangulation processing to obtain a fourth point cloud model; wherein, the third matching relationship includes the target second image and the first The pixel point matching relationship between the two neighborhood images, the fourth point cloud model and the second point cloud model have the same coordinate system and the same scale;
  • the merging parameter is acquired according to a fourth matching relationship between the point cloud included in the third point cloud model and the point cloud included in the fourth point cloud model.
  • both the second matching relationship and the third matching relationship are obtained according to the first matching relationship, where the first matching relationship is the target first image included in the target image pair and the pixel point matching relationship between the target second image.
  • the acquisition of the merging parameters according to the point cloud matching relationship between the point cloud included in the third point cloud model and the point cloud included in the fourth point cloud model includes:
  • the fourth matching relationship Acquiring the fourth matching relationship according to the first matching relationship, the second matching relationship, the third matching relationship, the fifth matching relationship and the sixth matching relationship, wherein the fifth matching relationship is the first neighborhood image
  • the matching relationship between the pixel points in the image and the point cloud in the third point cloud model, the sixth matching relationship is the pixel point in the second neighborhood image and the point cloud in the fourth point cloud model Matching relationship between point clouds;
  • the merging parameters are acquired according to the positions of the point clouds included in the fourth matching relationship in the corresponding coordinate system.
  • the acquisition of the merging parameters according to the fourth matching relationship between the point cloud included in the third point cloud model and the point cloud included in the fourth point cloud model includes:
  • the merge parameter is obtained by using an iterative nearest neighbor algorithm based on random sampling consistency.
  • the merging of the first point cloud model and the second point cloud model according to the merging parameters, after obtaining the target point cloud model further includes:
  • the combined parameters including:
  • the merging parameters are obtained.
  • the acquisition of the merging parameters according to the target pixel points and the target point cloud corresponding to the feature points includes:
  • the combination parameters are obtained by using a PnP algorithm based on random sampling consistency.
  • the merging of the first point cloud model and the second point cloud model according to the merging parameters, after obtaining the target point cloud model further includes:
  • the method also includes:
  • the camera positions of the first images in the first image set and the camera positions of the second images in the second image set are combined into the target point cloud model.
  • the method also includes:
  • the bundle adjustment optimization algorithm perform maximum likelihood estimation on the camera position of each of the first images in the target point cloud model, the new camera position of each of the second images, and the position of each point cloud, and obtain an estimation result.
  • the camera position of each of the first images, the camera position of each of the second images, and the position of each point cloud in the target point cloud model are respectively adjusted.
  • the method also includes:
  • an embodiment of the present disclosure provides a point cloud model processing device, including:
  • An image extraction module configured to acquire a target image pair and a first neighborhood image and a second neighborhood image corresponding to the target image pair according to the first image set and the second image set taken for the target scene; wherein, the The first image set includes a plurality of first images with a first visual effect; the second image set includes a plurality of second images with a second visual effect; the target image pair includes a target whose similarity satisfies a first condition The first image and the target second image; the first neighborhood image includes an image whose similarity with the target first image in the first image set satisfies a second condition; the second neighborhood image includes the An image whose similarity with the target second image in the second image set satisfies the third condition;
  • a parameter calculation module configured to obtain merging parameters according to the relationship between the target image pair, the first neighborhood image, the second neighborhood image, the first point cloud model, and the second point cloud model; wherein , the first point cloud model is a three-dimensional point cloud model reconstructed in advance according to the first image set; the second point cloud model is a three-dimensional point cloud model reconstructed in advance according to the second image set;
  • a merging module configured to merge the first point cloud model and the second point cloud model according to the merging parameters to obtain a target point cloud model.
  • an embodiment of the present disclosure provides an electronic device, including: a memory, a processor, and computer program instructions;
  • said memory is configured to store said computer program instructions
  • the processor is configured to execute the computer program instructions, and when the processor executes the computer program instructions, the method according to any one of the first aspect is implemented.
  • the embodiments of the present disclosure further provide a readable storage medium, including: a program
  • the electronic device When the program is executed by at least one processor of the electronic device, the electronic device implements the method according to any one of the first aspect.
  • an embodiment of the present disclosure further provides a program product, including: a computer program; the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the program product from the readable storage medium In the computer program, the at least one processor executes the computer program so that the electronic device implements the method according to any one of the first aspect.
  • Embodiments of the present disclosure provide a point cloud model processing method, device, and readable storage medium, by acquiring at least one set of target images from a first image set and a second image set that are shot for a target scene and have different visual effects pair and the first neighborhood image and the second neighborhood image corresponding to the target image pair; for each target image pair, according to the target image pair, the first neighborhood image and the second neighborhood image, the first point cloud model The relationship between the first point cloud model and the second point cloud model is calculated to obtain the merging parameters; according to the corresponding merging parameters of each group of target image pairs, the first point cloud model and the second point cloud model are merged to obtain the 3D image used to reconstruct the target scene The target point cloud model of the structure.
  • more original image information is used to participate in the merging of point cloud models. Compared with the method of merging directly based on point cloud models in the prior art, the consistency of the merged point cloud models obtained by this solution is higher.
  • FIG. 1 is a flowchart of a point cloud model processing method provided by an embodiment of the present disclosure
  • FIG. 2 is a flowchart of a point cloud model processing method provided by another embodiment of the present disclosure.
  • 3A to 3G are flowcharts of a point cloud model processing method provided by another embodiment of the present disclosure.
  • FIG. 4 is a schematic diagram of merging effects before global optimization and after global optimization using the point cloud model processing method provided by an embodiment of the present disclosure
  • FIG. 5 is a schematic structural diagram of a point cloud model processing device provided by an embodiment of the present disclosure.
  • Fig. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • SMF Structure from motion
  • the ICP algorithm is used to obtain the merging parameters of the two clusters of 3D point cloud models, so as to obtain the alignment between the two clusters of 3D point cloud models.
  • This method requires that the point cloud distribution is relatively uniform, and it cannot achieve a good merging effect for sparse point cloud models.
  • an embodiment of the present disclosure provides a method for processing a point cloud model.
  • the method selects a target image pair from image sets with different visual effects, and utilizes the difference between the target first image and the target second image included in the target image pair.
  • the image feature matching relationship between (2D-2D matching relationship) and the image feature matching relationship between the target image pair and other similar images (2D-2D matching relationship), and the relationship between the two cluster point cloud models before merging is established.
  • the spatial matching relationship and then solve the merging parameters.
  • this scheme uses more original image information to participate in the merging and alignment, the obtained merging parameters are more accurate, and the consistency of the merged point cloud model is stronger .
  • FIG. 1 is a flowchart of a point cloud model processing method provided by an embodiment of the present disclosure. As shown in Figure 1, the method of the present embodiment includes:
  • the first image set includes a plurality of first images with a first visual effect taken for the target scene; the second image set includes a plurality of second images with a second visual effect taken for the target scene.
  • the first visual effect is different from the second visual effect, and the difference between the first visual effect and the second visual effect may be caused by one or more factors such as time period, illumination, season, and weather.
  • the first image set includes a set of images taken for the target scene in sunny weather; the second image set includes a set of images taken for the target scene in cloudy weather.
  • the first set of images includes a set of images taken in summer for the target scene; the second set of images includes a set of images taken in winter for the target scene.
  • the first set of images includes a set of images taken in daytime for the target scene; the second set of images includes a set of images taken in night for the target scene.
  • the present disclosure is not limited to specific visual effects.
  • the number of target image pairs can be one group or multiple groups.
  • each group of target image pairs includes a target first image and a target second image whose similarity satisfies a first condition
  • the first image set includes the target first image
  • the second image set includes the target second image.
  • the first neighborhood image corresponding to the target image pair includes images whose similarity with the target first image in the first image set satisfies the second condition
  • the second neighborhood image corresponding to the target image pair includes the second image set and the target second image An image whose similarity satisfies the third condition.
  • the first condition, the second condition and the third condition can be respectively numericalized as: the first threshold, the second threshold and the third threshold; the numerical values of the first threshold, the second threshold and the third threshold can be the same, or can be Different; in practical applications, the values of the first threshold, the second threshold, and the third threshold can be set according to actual requirements, which is not limited in the embodiments of the present disclosure.
  • the first condition may also include: according to the similarity between the first image and the second image, sort and determine the top preset number of target image pairs in descending order; the second condition, The third condition is implemented similarly to the first condition.
  • the similarity between target image pairs may be jointly determined according to the similarity of image texture and the similarity of image semantics between each first image and each second image.
  • the coverage rate of the image for the target scene can also be considered, and the image pair with higher similarity and higher coverage rate can be selected as the target image pair.
  • the similarity between the first image of the target and other first images may be based on the offset between the shooting position of the first image of the target and the shooting positions of other first images, and the difference between the first image of the target and other first images.
  • the overlapping area between the images is jointly determined; wherein, the overlapping area can be jointly determined according to the shooting direction and shooting angle of the target first image, and the shooting directions and shooting angles of other first images.
  • Obtaining the target image pair from the first image set and the second image set can be achieved in any of the following ways:
  • a possible implementation manner includes: automatically extracting at least one set of target image pairs from the first image set and the second image set by using an image retrieval technology of deep learning.
  • the feature information of each first image can be obtained according to a pre-trained image retrieval model, wherein the feature information of the first image includes: image texture information and image semantic information of the first image; retrieve the model to obtain feature information of each second image, wherein the feature information of the second image includes: image texture information and image semantic information of the second image; for each first image, combine the feature information of the first image with the The feature information of each second image is matched, and the similarity between the first image and each second image is obtained; by performing the above process on each first image, any first image and any second image are obtained. The similarity between them; according to the similarity and the first condition, at least one group of target image pairs satisfying the first condition is acquired.
  • Another possible implementation method includes: using a manual selection method to select the target first image from the first image set, select the target second image from the second image set, and mark the target first image and the target second image as target image pair. If multiple sets of target image pairs are required, manually label them one by one.
  • obtaining the first neighborhood image and the second neighborhood image corresponding to the target image pair can be achieved in the following manner:
  • a pre-trained deep learning neural network model can be used to separately acquire the shooting position and shooting direction of the first image of the target, and the shooting positions and shooting directions of other first images in the first image set.
  • the offsets between the first image of the target and the shooting positions of other first images are obtained; according to the shooting direction of the first image of the target and the shooting positions of other first images
  • the angle between the shooting directions of the images and the shooting angle are used to obtain the size of the overlapping area between the first image of the target and other first images;
  • First image and other first image overlapping area size obtain the similarity between target first image and other first images respectively;
  • the first image of the second condition is the first neighborhood image.
  • the preset corresponding relationship can be queried according to the offset between the first image of the target and the shooting position of each first image and the size of the overlapping area, and the similarity between the first image of the target and each first image can be obtained, wherein, here
  • the preset correspondence referred to is the correspondence among the offset of the shooting position, the size of the overlapping area, and the similarity.
  • the implementation manner of acquiring the second neighborhood image is similar to the implementation manner of acquiring the first neighborhood image, and reference may be made to the specific description of acquiring the first neighborhood image. For the sake of brevity, details are not repeated here.
  • the first point cloud model can be a 3D point cloud model of the target scene reconstructed by using the SMF algorithm based on the first images in the first image set in advance; the second point cloud model can be based on the first images in the second image set in advance.
  • the SMF algorithm based on the first images in the first image set in advance
  • the second point cloud model can be based on the first images in the second image set in advance.
  • Each of the second images, and the 3D point cloud model of the target scene reconstructed using the SMF algorithm may also be used to reconstruct the first point cloud model and the second point cloud model respectively according to the first image set and the second image set, and this solution is not limited.
  • the target image pair can be one group or multiple groups.
  • the combination of the first point cloud model and the second point cloud model can be guided according to the obtained set of combination parameters.
  • each group of target image pairs can correspond to different regions in the target scene, and then the first point cloud model and the second point cloud model can be guided in a targeted manner according to the combination parameters obtained by each group of target image pairs.
  • the merging of corresponding areas in the point cloud model it should be noted that in the 3D scene reconstruction, the target scene (or the point cloud model before merging) is divided into multiple areas, and a set of merging parameters is calculated for each area separately, The merged point cloud model is more consistent.
  • the building when reconstructing the three-dimensional structure of a building, the building can be divided into: front, back, first side and second side. For the front of the building, obtain the target image pair 1; for the back of the building, obtain the target image pair 2; for the first side of the building, obtain the target image pair 3; for the second side of the building, obtain the target image pair 4.
  • the merging parameters corresponding to the target image pair 1 are used to merge the point cloud of the front of the building, and the merging parameters corresponding to the target image pair 2 are used to merge the building
  • the point cloud of the back area of the object is used to merge the point cloud of the first side of the building
  • the merging parameter corresponding to the target image pair 4 is used to merge the point cloud of the second side of the building.
  • Obtaining the corresponding merging parameters for each target image pair can be done in any of the following ways:
  • a possible implementation method may adopt an automatic method to obtain the merge parameters, which may specifically include the following steps:
  • Step 1 Obtain the first matching relationship according to the feature matching method based on deep learning; and obtain the second matching relationship and the third matching relationship according to the pixels included in the first matching relationship.
  • the first matching relationship is the pixel point matching relationship between the target first image and the target second image
  • the second matching relationship is the pixel point matching relationship between the target first image and the first neighborhood image
  • the third matching The relationship is the pixel point matching relationship between the target second image and the second neighborhood image.
  • the method of feature matching based on deep learning includes: using deep learning method to pre-train an image matching model that meets the requirements, and using the image matching model to obtain pixel point matching results between images.
  • the image matching model first performs hotspot learning on one of the images (assumed to be image A), and determines the hotspots according to the response values of different regions, where the higher the response value, the region is The higher the probability of the hotspot area, the lower the response value, which means the lower the probability value of the area as a hotspot area; the feature information of the pixels in the hotspot area is extracted by using the image matching model, and the feature information includes image texture information and image semantic information; Then, the image matching model performs matching in other images according to the image information and texture information of the pixels in the hotspot area of image A, so as to obtain the pixel point matching relationship between the images.
  • the first matching relationship includes A1 and B1, A2 and B2...An and Bn matching relationship between.
  • the matching is performed in each pixel included in the first neighborhood image, and the second matching relationship is generated based on the matching result, and the second matching relationship includes A1 and C1 , the matching relationship between A2 and C2...Am and Cm, wherein C1, C2...Cm are the pixels in the first neighborhood image.
  • the third matching relationship includes B1 and D1, B2 and D2... ...the matching relationship between Bi and Di, wherein D1, D2...Di are the pixels of the second neighborhood image.
  • n, m, and i are all integers greater than or equal to 1, and m is less than or equal to n, and i is less than or equal to n.
  • first matching relationship, second matching relationship and third matching relationship are all matching relationships between two-dimensional (2D) pixel points.
  • the number of the second matching relationship and the third matching relationship can be multiple, the number of the second matching relationship is the same as the number of the first neighborhood images, and the number of the third matching relationship is the same as the number of the second neighborhood images.
  • Step 2 according to the second matching relationship and the camera position of each first neighborhood image, perform feature triangulation processing to obtain the third point cloud model; according to the third matching relationship and the camera position of each second neighborhood image, perform feature Triangulation processing to obtain the fourth point cloud model.
  • the third point cloud model has the same coordinate system and the same scale as the first point cloud model;
  • the fourth point cloud model has the same coordinate system and the same scale as the second point cloud model.
  • the pixels in the target first image included in the second matching relationship are a subset of the pixels in the target first image included in the first matching relationship, therefore, the pixel points obtained according to the second matching relationship
  • the third point cloud model is equivalent to a part of the first point cloud model; similarly, the pixels in the target second image included in the third matching relationship are a subset of the pixels in the target second image included in the first matching relationship, Therefore, the fourth point cloud model obtained according to the third matching relationship is equivalent to a part of the second point cloud model.
  • the third point cloud model has the same coordinate system and scale as the first point cloud model
  • the fourth point cloud model has the same coordinate system and scale as the second point cloud model
  • Step 3 according to the fourth matching relationship between the point cloud included in the third point cloud model and the point cloud included in the fourth point cloud model, the merging parameters are obtained.
  • the fourth matching relationship is obtained, wherein the fourth matching relationship is the points included in the third point cloud model
  • the matching relationship between the cloud and the point cloud included in the fourth point cloud model can be understood as a 3D-3D correspondence.
  • the fifth matching relationship is the matching relationship between the pixel point in the first neighborhood image and the point cloud in the third point cloud model
  • the sixth matching relationship is the pixel point in the second neighborhood image and the fourth point
  • the matching relationship between point clouds in the cloud model According to the position of each point cloud included in the fourth matching relationship in the corresponding coordinate system, the iterative nearest neighbor algorithm based on random sampling consistency (RANSAC-ICP) is used to obtain the merging parameters.
  • RANSAC-ICP random sampling consistency
  • the matching relationship between each first image in the first image set and the first point cloud model is obtained when the first point cloud model is created.
  • the matching relationship between each second image in the second image set and the second point cloud model is obtained when the second point cloud model is created.
  • this solution uses the RANSAC-ICP algorithm to remove the mismatching and ensure the correct matching. Among them, the incorrectly matched point clouds are called outliers, and the correctly matched point clouds are called inliers.
  • M is a positive integer greater than or equal to 3
  • W is a positive integer greater than or equal to 1.
  • Another possible implementation method can be implemented manually. It should be noted that the "manual method" referred to here means that the information obtained in some steps is realized manually, and not all steps are manually implemented. way to achieve. For example, in the implementation manner shown below, the matching relationship among the target pixel points, target point cloud and feature points in the target scene obtained in step 1 is obtained based on manual annotation.
  • Step 1 Obtain the target pixel points corresponding to the feature points of the pre-marked target scene in the target image pair, the first neighborhood image and the second neighborhood image respectively, and obtain the feature points of the above target scene respectively in the first point cloud model and the corresponding target point cloud in the second point cloud model.
  • the above-mentioned pre-marked target pixel points and target point cloud may be manually marked according to feature points of the target scene. It should be understood that there is a matching relationship among the target pixel points, the target point cloud, and the feature points in the target scene, and the matching relationship can be understood as a 2D-3D correspondence relationship.
  • Step 2 According to the above target pixel points and target feature points, merge parameters are obtained.
  • a possible implementation method according to the positions of the target pixel points in the image and the position of the target point cloud in the point cloud model, adopt the PnP algorithm (RANSAC-PnP) based on random sampling consistency to obtain the merged parameter.
  • RNSAC-PnP PnP algorithm
  • this solution uses RANSAC-PnP to remove mismatches and ensure correct matches.
  • the incorrectly matched point cloud (or pixel point) is called an outlier point
  • the correctly matched point cloud (or pixel point) is called an inlier point.
  • the point clouds of the corresponding regions in the first point cloud model or the second point cloud model are merged, for example, scaling processing, rotation Process and translate one or more of the processes.
  • the point cloud model processing method provided in this embodiment obtains at least one set of target image pairs and the first image pairs corresponding to the target image pairs from the first image set and the second image set that are shot for the target scene and have different visual effects. Neighborhood images and second neighborhood images; for each group of target image pairs, according to the target image pair, the first neighborhood image and the second neighborhood image, the first point cloud model and the second point cloud model According to the corresponding combination parameters of each group of target image pairs, the first point cloud model and the second point cloud model are combined to obtain the target point cloud model used to reconstruct the three-dimensional structure of the target scene.
  • more original image information is used to participate in the merging of point cloud models. Compared with the method of merging directly based on point cloud models in the prior art, the consistency of the merged point cloud models obtained by this solution is higher.
  • Fig. 2 is a flowchart of a point cloud model processing method provided by another embodiment of the present disclosure. Referring to FIG. 2, on the basis of the embodiment shown in FIG. 1, after S103, it may also include:
  • the merging parameter of the target image pair is to perform one or more of scaling processing, rotation processing and translation processing on each camera position in the corresponding area in the coordinate system of the second point cloud model.
  • the shooting positions of the images in different image sets are aligned to the coordinate system of the target point cloud model, so as to provide a basis for subsequent other image processing.
  • steps S105 and S106 may also be included:
  • S105 Perform maximum likelihood estimation on the camera position of each first image, the new camera position of each second image, and the position of each point cloud in the coordinate system of the target point cloud model according to the cluster adjustment optimization algorithm, and obtain an estimation result.
  • the cluster adjustment optimization algorithm is used to perform global optimization to reduce the reprojection error from the 3D point to the 2D image point, so that the adjusted target point cloud model is more consistent.
  • the method may further include: S107. Reconstructing the three-dimensional structure of the target scene according to the target point cloud model.
  • 3A-3G are flowcharts of a point cloud model processing method provided by another embodiment of the present disclosure.
  • the first image set includes a set of images taken during the day for the target scene
  • the second image set includes a set of images taken about the target scene at night for example.
  • the first image set takes a daytime image database as an example, and the daytime image database includes a plurality of images taken during the daytime for the target scene, and the images in the daytime image database can be called daytime image samples; the second image set is represented by The night image database is an example.
  • the night image database includes multiple images taken at night for the target scene.
  • the images in the night image database can be called night image samples;
  • the point cloud model reconstructed in advance according to the daytime image database is a 3D point cloud model A
  • the point cloud model reconstructed in advance according to the night image database is the 3D point cloud model B.
  • the daytime image sample R1 and the nighttime image sample S1 are a group of target image pairs, and the daytime image samples R2, R3, and R4 are the first of R1.
  • a neighborhood image, night image samples S2 , S3 and S4 are the second neighborhood images of S1 .
  • the solid six-pointed star and solid circle connected by the arrows at both ends of the dotted line are a set of matching pixels, wherein the solid six-pointed star represents the pixel point with a matching relationship in the daytime image sample R1, and the solid circle represents the night Pixels with a matching relationship in the image sample S1.
  • pixels that do not have a matching relationship in the daytime image sample R1 and the nighttime image sample S1 are not shown by marks.
  • the hollow hexagonal star represents the point cloud in the 3D point cloud model A'
  • the hollow circle represents the point cloud in the 3D point cloud model B'
  • the hollow hexagon and the hollow circle connected by the arrows at both ends of the dotted line are one Group point clouds with matching relationships.
  • the feature points in the artificially selected daytime image samples R1, R2, R3 and R4 can be associated with the corresponding point clouds in the 3D point cloud model A and/or the 3D point cloud model B;
  • the feature points in the selected night image samples S1 , S2 , S3 and S4 are associated with the corresponding point clouds in the 3D point cloud model A and/or the 3D point cloud model B, thereby obtaining the 2D-3D correspondence.
  • the feature points manually selected in the daytime image samples and the feature points manually selected in the night image samples usually point to the same feature point in the target scene. Pointing to the same feature point in the target scene can be determined from the points whose three-dimensional structure is recovered from the 3D point cloud model A or the 3D point cloud model B.
  • the hollow hexagonal star represents the point cloud in the 3D point cloud model A
  • the hollow circle represents the point cloud in the 3D point cloud model B
  • the hollow hexagons and hollow circles connected by the arrows at both ends of the dotted line segment form a group with
  • the hollow circle and the solid circle connected by the arrows at both ends of the dotted line segment are a group of point clouds and pixels with a matching relationship
  • the solid hexagons and solid circles connected by the arrows at both ends of the dotted line segment are a group of matching relationship pixels
  • the hollow hexagons and solid hexagons connected by the arrows at both ends of the dotted line segment are a set of point clouds and pixel points with a matching relationship.
  • FIG. 3E only shows the matching relationship between some pixel points and the point cloud.
  • RANSAC-ICP to calculate the 3D-3D correspondence obtained by the method shown in Figure 3D
  • RANSAC-PnP to calculate the 2D-3D correspondence obtained by the method shown in Figure 3E to obtain a 3D point cloud
  • the merging parameters between model A and 3D point cloud model B, the merging parameters are expressed as [s, r, t], where s represents the scaling scale, r represents the rotation parameter, and t represents the translation parameter; and according to the merging parameter [s, r, t] to merge point 3D cloud model A and 3D point cloud model B.
  • each point cloud of the 3D point cloud model A is carried out by using the merged parameters [s, r, t] obtained from the solution, and each point cloud of the converted 3D point cloud model A is compared with the 3D
  • Each point cloud of point cloud model B performs a corresponding merging operation, thereby aligning point cloud model A to point cloud model B, and merging the point clouds in 3D point cloud model A and 3D point cloud model B to obtain a new point
  • the cloud model is the target point cloud model described in the foregoing embodiments.
  • the coordinate system conversion is performed on the 3D point cloud model A.
  • the coordinate system conversion can also be performed on the 3D point cloud model B according to the merging parameters, and the converted 3D point cloud model B is merged into the 3D point cloud.
  • the coordinate system of model A In the coordinate system of model A.
  • the camera position of each daytime image sample in the daytime image database can also be merged into the coordinate system of the 3D point cloud model B according to the merging parameters, and the camera position of each night image sample in the nighttime image database can be merged into the three-dimensional Coordinate system of point cloud model B.
  • the above obtained 3D-3D correspondence or 2D-3D correspondence may be incorporated into the target point cloud model.
  • the hollow four-pointed star represents the point cloud in the target point cloud model after merging
  • the hollow four-pointed star and the solid six-pointed star connected by the arrows at both ends of the dotted line represent point clouds and pixels with a matching relationship
  • the points at both ends of the dotted line segment The hollow four-pointed star and solid circle connected by arrows indicate the point cloud and pixel points with matching relationship.
  • the line segment shown in FIG. 3G embodies the above-mentioned 3D-3D correspondence relationship or 2D-3D correspondence relationship.
  • the target point cloud model obtained according to Fig. In order to reduce the reprojection error from the reconstructed 3D point to the 2D image point and improve the consistency of the target point cloud model, in the target point cloud model obtained according to Fig.
  • the camera position, the camera position of each night image sample, and the position of each point cloud are globally optimized for the target point cloud model obtained in Figure 3F using the bundle adjustment optimization algorithm.
  • FIG. 4 is a schematic diagram of the combined effect before and after global optimization using the point cloud model processing method provided by the embodiment of the present disclosure, wherein (a) in FIG. The point cloud model processing method provided by the disclosed embodiment, but a schematic diagram of the merging effect without global optimization, (b) in Figure 4 is the point cloud model processing method provided by the disclosed embodiment, and the merging effect after global optimization schematic diagram. Comparing the two situations shown in (a) and (b) in Figure 4, and referring to (b) in Figure 4, there are many new points obtained by the cluster adjustment optimization algorithm on both sides of the building Cloud, and the position of the point cloud model shown in (b) in Figure 4 is also fine-tuned according to the estimation results, making the point cloud model more accurate. Therefore, it can be seen that the target point cloud model after global optimization using the cluster adjustment optimization algorithm contains more new point clouds, and the position of the optimized point cloud model is more accurate, and the obtained point cloud model is more consistent.
  • Fig. 5 is a schematic structural diagram of a point cloud model processing device provided by an embodiment of the present disclosure.
  • the point cloud model processing device 500 provided in this embodiment includes: an image extraction module 501 , a parameter calculation module 502 , and a combination module 503 .
  • the image extraction module 501 is configured to obtain the target image pair and the first neighborhood image and the second neighborhood image corresponding to the target image pair according to the first image set and the second image set taken for the target scene; wherein , the first image set includes a plurality of first images with a first visual effect; the second image set includes a plurality of second images with a second visual effect; the target image pair includes a similarity satisfying the first Conditional target first image and target second image; the first neighborhood image includes an image whose similarity with the target first image in the first image set satisfies a second condition; the second neighborhood image Including images whose similarities between the second image set and the target second image satisfy a third condition.
  • a parameter calculation module 502 configured to obtain merging parameters according to the relationship between the target image pair, the first neighborhood image, the second neighborhood image, the first point cloud model, and the second point cloud model;
  • the first point cloud model is a 3D point cloud model reconstructed in advance according to the first image set;
  • the second point cloud model is a 3D point cloud model reconstructed in advance according to the second image set.
  • the merging module 503 is configured to merge the first point cloud model and the second point cloud model according to the merging parameters to obtain a target point cloud model.
  • the image extraction module 501 is specifically configured to input the first image set and the second image set into a pre-trained image retrieval model, and obtain the target image output by the image retrieval model pair, and the first neighborhood image and the second neighborhood image corresponding to the target image pair.
  • the parameter calculation module 502 is specifically configured to perform feature triangulation processing according to the second matching relationship and the camera position corresponding to the first neighborhood image to obtain a third point cloud model; wherein, the The second matching relationship includes the pixel point matching relationship between the first image of the target and the first neighborhood image, and the coordinate system of the third point cloud model is the same as that of the first point cloud model and the scale is Same; according to the third matching relationship and the camera position corresponding to the second neighborhood image, perform feature triangulation processing to obtain a fourth point cloud model; wherein, the third matching relationship includes the second image of the target and the The pixel point matching relationship between the second neighborhood image, the fourth point cloud model and the second point cloud model have the same coordinate system and the same scale; according to the points included in the third point cloud model The fourth matching relationship between the cloud and the point cloud included in the fourth point cloud model is to obtain the combination parameter.
  • both the second matching relationship and the third matching relationship are obtained according to the first matching relationship, where the first matching relationship is the target first image included in the target image pair and the pixel point matching relationship between the target second image.
  • the parameter calculation module 502 is specifically configured to obtain the fourth matching relationship according to the first matching relationship, the second matching relationship, the third matching relationship, the fifth matching relationship and the sixth matching relationship, wherein , the fifth matching relationship is the matching relationship between the pixel points in the first neighborhood image and the point cloud in the third point cloud model, and the sixth matching relationship is the matching relationship between the pixels in the second neighborhood image
  • the merging parameter is obtained.
  • the parameter calculation module 502 is specifically configured to use iteration based on random sampling consistency according to the fourth matching relationship and the position of each point cloud included in the fourth matching relationship in the corresponding coordinate system The nearest neighbor algorithm is used to obtain the merging parameters.
  • the merging module 503 is further configured to merge the first point cloud model and the second point cloud model according to the merging parameters, and after obtaining the target point cloud model, combine the second The four matching relationships are merged into the target point cloud model.
  • the parameter calculation module 502 is specifically configured to obtain the pre-marked feature points in the target scene in the target image pair, the first neighborhood image and the second neighborhood image respectively.
  • the corresponding target pixel points in and the pre-labeled feature points respectively correspond to the point clouds in the first point cloud model and the second point cloud model; according to the target pixel points and target points corresponding to the feature points point cloud, to obtain the merged parameters.
  • the parameter calculation module 502 is specifically configured to use random sampling consistency based on the position of each target pixel in the image and the position of the target point cloud in the point cloud model. The PnP algorithm to obtain the combination parameters.
  • the merging module 503 is also used to merge the first point cloud model and the second point cloud model according to the merging parameters, and after obtaining the target point cloud model, merge the The corresponding relationship among the feature points, the target pixel points and the target point cloud is merged into the target point cloud model.
  • the merging module 503 is further configured to merge the camera positions of each first image in the first image set and the camera positions of each second image in the second image set into In the target point cloud model.
  • the point cloud model processing apparatus 500 further includes: a point cloud model optimization module 504 .
  • the point cloud model optimization module 504 is configured to adjust the camera position of each first image, the new camera position of each second image, and each point cloud in the target point cloud model according to the bundle adjustment optimization algorithm. The maximum likelihood estimation is performed on the position, and the estimation result is obtained; according to the estimation result, the camera position of each of the first images in the target point cloud model, the camera position of each of the second images, and the position of each point cloud The positions are adjusted separately.
  • the point cloud model processing device provided in this embodiment can be used to implement the technical solutions of any of the foregoing method embodiments, and its implementation principles and technical effects are similar, and reference can be made to the descriptions of the foregoing embodiments, which will not be repeated here.
  • FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • an electronic device 600 provided in this embodiment includes: a memory 601 and a processor 602 .
  • the memory 601 may be an independent physical unit, and may be connected to the processor 602 through a bus 603 .
  • the memory 601 and the processor 602 may also be integrated together, implemented by hardware, and the like.
  • the memory 601 is used to store program instructions, and the processor 602 invokes the program instructions to execute the operations of any one of the above method embodiments.
  • the foregoing electronic device 600 may also include only the processor 602 .
  • the memory 601 for storing programs is located outside the electronic device 600, and the processor 602 is connected to the memory through circuits/wires for reading and executing the programs stored in the memory.
  • the processor 602 may be a central processing unit (central processing unit, CPU), a network processor (network processor, NP) or a combination of CPU and NP.
  • CPU central processing unit
  • NP network processor
  • the processor 602 may further include a hardware chip.
  • the aforementioned hardware chip may be an application-specific integrated circuit (application-specific integrated circuit, ASIC), a programmable logic device (programmable logic device, PLD) or a combination thereof.
  • the aforementioned PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL) or any combination thereof.
  • the memory 601 may include a volatile memory (volatile memory), such as a random-access memory (random-access memory, RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory) ), a hard disk (hard disk drive, HDD) or a solid-state drive (solid-state drive, SSD); the memory can also include a combination of the above-mentioned types of memory.
  • volatile memory such as a random-access memory (random-access memory, RAM
  • non-volatile memory such as a flash memory (flash memory)
  • HDD hard disk drive
  • solid-state drive solid-state drive
  • the present disclosure also provides a computer-readable storage medium.
  • the computer-readable storage medium includes computer program instructions. When the computer program instructions are executed by at least one processor of the electronic device, the technical solution of any one of the above method embodiments is executed. .
  • the present disclosure also provides a program product, the program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of the electronic device can read the computer program from the readable storage medium.
  • the computer program, the at least one processor executes the computer program so that the electronic device executes the technical solution of any one of the above method embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Library & Information Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Hardware Design (AREA)
  • Architecture (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种点云模型处理方法，包括：从针对目标场景拍摄的，且具有不同视觉效果的第一图像集和第二图像集中，获取目标图像对以及目标图像对对应的第一邻域图像和第二邻域图像；根据目标图像对、第一邻域图像以及第二邻域图像、第一点云模型和第二点云模型之间的关系，计算获得合并参数；根据合并参数，对第一点云模型和第二点云模型进行合并，获取用于重建目标场景的三维结构的目标点云模型。还公开了一种点云模型处理装置、电子设备及可读存储介质。

Description

点云模型处理方法、装置及可读存储介质
相关申请的交叉引用
本申请要求于2021年05月13日提交的，申请号为202110524280.6、发明名称为“点云模型处理方法、装置及可读存储介质”的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开涉及图像处理技术领域，尤其涉及一种点云模型处理方法、装置及可读存储介质。
背景技术
由于图像数据获取速度快且成本低，因此，基于图像的三维重建技术在测量、虚拟现实、电影娱乐、高精度地图等各个领域被广泛应用。在三维重建技术中，通过拍摄物体的多帧图像或者视频序列，进行三维重建获得多簇三维点云模型，之后对重建的多簇三维点云模型进行合并，根据合并后的三维点云模型恢复出物体的三维结构。
现有技术中，通常根据重建的多簇三维点云模型，采用迭代最近邻算法(Iterative Closest Point，ICP)获取合并参数，并根据合并参数对多簇三维点云模型进行合并。上述方式是直接根据点云进行坐标对齐，获取的合并后的点云模型精确度较低。
发明内容
为了解决上述技术问题或者至少部分地解决上述技术问题，本公开提供了一种点云模型处理方法、装置及可读存储介质。
第一方面，本公开实施例提供了一种点云模型处理方法，包括：
根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及所述目标图像对对应的第一邻域图像和第二邻域图像；其中，所述第一图像集包括具有第一视觉效果的多幅第一图像；所述第二图像集包括具有第二视觉效果的多幅第二图像；所述目标图像对包括相似度满足第一条件的目标第一图像和目标第二图像；所述第一邻域图像包括所述第一图像集中与所述目标第一图像的相似度满足第二条件的图像；所述第二邻域图像包括所述第二图像集中与所述目标第二图像的相似度满足第三条件的图像；
根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型、以及第二点云模型之间的关系，获取合并参数；其中，所述第一点云模型包括预先根据所述第 一图像集重建的三维点云模型；所述第二点云模型包括预先根据所述第二图像集重建的三维点云模型；
根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型。
在一些可能的设计中，所述根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及述目标图像对对应的第一邻域图像和第二邻域图像，包括：
将所述第一图像集和所述第二图像集输入至预先训练好的图像检索模型中，获取所述图像检索模型输出的目标图像对、以及所述目标图像对对应的第一邻域图像和第二邻域图像。
在一些可能的设计中，所述根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型、和第二点云模型之间的关系，获取合并参数，包括：
根据第二匹配关系、所述第一邻域图像对应的相机位置，进行特征三角化处理，获取第三点云模型；其中，所述第二匹配关系包括所述目标第一图像与所述第一邻域图像两者之间的像素点匹配关系，所述第三点云模型与所述第一点云模型的坐标系相同且尺度相同；
根据第三匹配关系及所述第二邻域图像对应的相机位置，进行特征三角化处理，获取第四点云模型；其中，所述第三匹配关系包括所述目标第二图像与所述第二邻域图像两者之间的像素点匹配关系，所述第四点云模型与所述第二点云模型的坐标系相同且尺度相同；
根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的第四匹配关系，获取所述合并参数。
在一些可能的设计中，所述第二匹配关系和所述第三匹配关系均是根据第一匹配关系获得的，其中，所述第一匹配关系为所述目标图像对包括的目标第一图像和目标第二图像之间的像素点匹配关系。
在一些可能的设计中，所述根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的点云匹配关系，获取所述合并参数，包括：
根据第一匹配关系、第二匹配关系、第三匹配关系、第五匹配关系以及第六匹配关系，获取所述第四匹配关系，其中，所述第五匹配关系为所述第一邻域图像中的像素点与所述第三点云模型中的点云之间的匹配关系，所述第六匹配关系为所述第二邻域图像中的像素点与所述第四点云模型中的点云之间的匹配关系；
根据所述第四匹配关系包含的各点云在相应坐标系中的位置，获取所述合并参数。
在一些可能的设计中，所述根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的第四匹配关系，获取所述合并参数，包括：
根据所述第四匹配关系以及所述第四匹配关系包含的各点云在相应坐标系中的位置， 采用基于随机采样一致性的迭代最近邻算法，获取所述合并参数。
在一些可能的设计中，所述根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型之后，还包括：
将所述第四匹配关系，合并至所述目标点云模型中。
在一些可能的设计中，所述根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型和第二点云模型之间的关系，获取合并参数，包括：
获取预先标注的所述目标场景中的特征点分别在所述目标图像对、所述第一邻域图像以及所述第二邻域图像中对应的目标像素点、及预先标注的所述特征点分别在所述第一点云模型和所述第二点云模型中对应的点云；
根据所述特征点对应的目标像素点及目标点云，获取合并参数。
在一些可能的设计中，所述根据所述特征点对应的目标像素点及目标点云，获取所述合并参数，包括：
根据各所述目标像素点在所属图像中的位置，及所述目标点云在所属点云模型中的位置，采用基于随机采样一致性的PnP算法，获取所述合并参数。
在一些可能的设计中，所述根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型之后，还包括：
将所述特征点、所述目标像素点以及所述目标点云三者之间的对应关系，合并至所述目标点云模型中。
在一些可能的设计中，所述方法还包括：
根据所述合并参数，将第一图像集中各第一图像的相机位置以及第二图像集中各第二图像的相机位置合并至所述目标点云模型中。
在一些可能的设计中，所述方法还包括：
根据集束调整优化算法，对所述目标点云模型中各所述第一图像的相机位置、各所述第二图像的新的相机位置以及各点云的位置进行最大似然估计，获取估计结果；
根据所述估计结果，对所述目标点云模型中各所述第一图像的相机位置、各所述第二图像的相机位置以及各点云的位置分别进行调整。
在一些可能的设计中，所述方法还包括：
根据所述目标点云模型重建所述目标场景的三维结构。
第二方面，本公开实施例提供一种点云模型处理装置，包括：
图像提取模块，用于根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及所述目标图像对对应的第一邻域图像和第二邻域图像；其中，所述第一图像集包括具有第一视觉效果的多幅第一图像；所述第二图像集包括具有第二视觉效果的多幅第二 图像；所述目标图像对包括相似度满足第一条件的目标第一图像和目标第二图像；所述第一邻域图像包括所述第一图像集中与所述目标第一图像的相似度满足第二条件的图像；所述第二邻域图像包括所述第二图像集中与所述目标第二图像的相似度满足第三条件的图像；
参数计算模块，用于根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型和第二点云模型之间的关系，获取合并参数；其中，所述第一点云模型是预先根据所述第一图像集重建的三维点云模型；所述第二点云模型是预先根据所述第二图像集重建的三维点云模型；
合并模块，用于根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型。
第三方面，本公开实施例提供一种电子设备，包括：存储器、处理器以及计算机程序指令；
所述存储器被配置为存储所述计算机程序指令；
所述处理器被配置为执行所述计算机程序指令，当所述处理器执行所述计算机程序指令时，实现如第一方面任一项所述的方法。
第四方面，本公开实施例还提供一种可读存储介质，包括：程序；
所述程序被电子设备的至少一个处理器执行时，所述电子设备实现如第一方面任一项所述的方法。
第五方面，本公开实施例还提供一种程序产品，包括：计算机程序；所述计算机程序存储在可读存储介质中，电子设备的至少一个处理器可以从所述可读存储介质中读取所述计算机程序，所述至少一个处理器执行所述计算机程序使得所述电子设备实现如第一方面任一项所述的方法。
本公开实施例提供一种点云模型处理方法、装置及可读存储介质，通过从针对目标场景拍摄的，且具有不同视觉效果的第一图像集和第二图像集中，获取至少一组目标图像对以及目标图像对对应的第一邻域图像和第二邻域图像；针对每组目标图像对，根据所述目标图像对、第一邻域图像以及第二邻域图像、第一点云模型和第二点云模型之间的关系，计算获得合并参数；根据各组目标图像对对应的合并参数，对第一点云模型和第二点云模型进行合并，获取用于重建目标场景的三维结构的目标点云模型。本实施例通过使用更多的原始图像信息参与点云模型的合并，相对于现有技术中直接根据点云模型进行合并的方式，本方案获得的合并后的点云模型的一致性更高。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开一实施例提供的点云模型处理方法的流程图；
图2为本公开另一实施例提供的点云模型处理方法的流程图；
图3A至图3G为本公开另一实施例提供的点云模型处理方法的流程图；
图4为采用本公开实施例提供的点云模型处理方法进行全局优化前和进行全局优化后分别对应的合并效果示意图；
图5为本公开一实施例提供的点云模型处理装置的结构示意图；
图6为本公开一实施例提供的电子设备的结构示意图。
具体实施方式
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。
根据多幅图像或者视频序列中自动地恢复相机参数和场景的三维结构，被称为运动恢复结构(Structure from motion，SMF)。在进行大规模场景重建时，SMF算法得到广泛应用。然而，SMF算法强烈依赖于图像之间特征点的匹配，当图像采集来自不同时段、不同季节、不同光照等时，图像的视觉效果差异较大，进而导致图像特征点匹配的可靠度降低。因此，提出先对采集的图像进行划分，然后根据划分后的图像分别重建产生三维点云模型，再对多簇三维点云模型进行合并。
采用ICP算法获取两簇三维点云模型的合并参数，从而获得两簇三维点云模型之间的对齐。这种方式要求点云分布相对均匀，针对稀疏点云模型无法取得较好的合并效果。
在一些情况下，还有先通过相机位置形成的采集轨迹在生成点云模型时进行粗对齐的方式，之后，再利用ICP算法获取合并参数并进行对齐。然而利用采集轨迹进行粗对齐要求采集视角相对一致，导致应用场景受限。
针对上述问题，本公开实施例提供一种点云模型处理方法，该方法通过从视觉效果不 同的图像集合中选取目标图像对，并利用目标图像对包括的目标第一图像和目标第二图像之间的图像特征匹配关系(2D-2D的匹配关系)以及目标图像对与其他相近的图像之间的图像特征匹配关系(2D-2D的匹配关系)，建立合并前的两簇点云模型之间的空间匹配关系，进而求解合并参数。相对于直接根据两簇点云模型计算合并参数并进行合并的方式，本方案通过更多的原始图像信息参与合并对齐，获得的合并参数更为精准，合并后的点云模型的一致性更强。
图1为本公开一实施例提供的点云模型处理方法的流程图。如图1所示，本实施例的方法包括：
S101、根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及所述目标图像对对应的第一邻域图像和第二邻域图像。
第一图像集包括针对目标场景拍摄的具有第一视觉效果的多幅第一图像；第二图像集包括针对目标场景拍摄的具有第二视觉效果的多幅第二图像。其中，第一视觉效果和第二视觉效果不同，第一视觉效果和第二视觉效果的不同可以是由于时段、光照、季节、天气等一个或多个因素导致的。
示例性地，第一图像集包括针对目标场景在晴朗天气时拍摄的图像的集合；第二图像集包括针对目标场景在阴雨天气时拍摄的图像的集合。或者，第一图像集包括针对目标场景在夏天拍摄的图像的集合；第二图像集包括针对目标场景在冬天拍摄的图像的集合。或者，第一图像集包括针对目标场景在白天拍摄的图像的集合；第二图像集包括针对目标场景在夜晚拍摄的图像的集合。本公开不对具体的视觉效果进行限制。
本方案中，目标图像对的数量可以为一组，也可以为多组。
具体地，每组目标图像对包括相似度满足第一条件的目标第一图像和目标第二图像，第一图像集包括目标第一图像，第二图像集包括目标第二图像。目标图像对对应的第一邻域图像包括第一图像集中与目标第一图像的相似度满足第二条件的图像；目标图像对对应的第二邻域图像包括第二图像集中与目标第二图像的相似度满足第三条件的图像。
其中，第一条件、第二条件以及第三条件可以分别数值化为：第一阈值、第二阈值以及第三阈值；第一阈值、第二阈值以及第三阈值的数值大小可以相同，也可以不同；在实际应用中，第一阈值、第二阈值以及第三阈值的数值大小可根据实际需求设置，本公开实施例对此不作限制。或者，第一条件也可以包括：根据第一图像与第二图像之间的相似度按照由高到低的顺序进行排序并确定的排名靠前的预设数量组目标图像对；第二条件、第三条件与第一条件的实现方式类似。
可选地，目标图像对之间的相似度可以是根据各第一图像和各第二图像之间的图像纹理的相似度以及图像语义的相似度共同确定的。在实际应用中，确定目标图像对时，还可 以考虑图像针对目标场景的覆盖率，可以选取相似度较高，且覆盖率较高的图像对作为目标图像对。
目标第一图像与其他第一图像之间的相似度可以是根据目标第一图像的拍摄位置和其他第一图像之间的拍摄位置之间的偏移量、以及目标第一图像与其他第一图像之间的重叠区域共同确定的；其中，重叠区域可根据目标第一图像的拍摄方向和拍摄角度、以及其他第一图像的拍摄方向和拍摄角度共同确定。
从第一图像集和第二图像集中，获取目标图像对可通过下述任一种方式实现：
一种可能的实现方式包括：利用深度学习的图像检索技术从第一图像集和第二图像集中自动提取至少一组目标图像对。具体地，可以根据预先训练好的图像检索模型，获取各第一图像的特征信息，其中，第一图像的特征信息包括：第一图像的图像纹理信息和图像语义信息；根据预先训练好的图像检索模型，获取各第二图像的特征信息，其中，第二图像的特征信息包括：第二图像的图像纹理信息和图像语义信息；针对每幅第一图像，将第一图像的特征信息分别与各第二图像的特征信息进行匹配，获取第一图像分别与各第二图像之间的相似度；通过对每幅第一图像执行上述过程，获得任意第一图像与任意第二图像，两两之间的相似度；根据相似度以及第一条件，获取满足第一条件的至少一组目标图像对。
另一种可能的实现方式包括：利用人工选取的方法，从第一图像集中选取目标第一图像，从第二图像集中选取目标第二图像，并将目标第一图像和目标第二图像标记为目标图像对。若需要多组目标图像对，人工一一标注即可。
根据目标图像对，获取目标图像对对应的第一邻域图像和第二邻域图像可通过下述方式实现：
首先，可以利用预先训练好的深度学习的神经网络模型，分别获取目标第一图像的拍摄位置以及拍摄方向，以及第一图像集中的其他第一图像的拍摄位置以及拍摄方向。
接着，根据目标第一图像的拍摄位置和其他第一图像的拍摄位置，获取目标第一图像分别与其他第一图像的拍摄位置的偏移量；根据目标第一图像的拍摄方向和其他第一图像的拍摄方向之间的夹角，以及拍摄角度，获取目标第一图像与其他第一图像的重叠区域大小；根据上述目标第一图像分别与其他第一图像的拍摄位置的偏移量、目标第一图像和其他第一图像重叠区域大小，获取目标第一图像分别与其他第一图像之间的相似度；并根据目标第一图像分别与其他第一图像之间的相似度，确定满足第二条件的第一图像为第一邻域图像。
可选地，可根据目标第一图像与各第一图像的拍摄位置的偏移量以及重叠区域大小查询预设对应关系，获取目标第一图像分别与各第一图像的相似度，其中，这里所指的预设对应关系为拍摄位置的偏移量、重叠区域大小以及相似度三者之间的对应关系。
获取第二邻域图像的实现方式与获取第一邻域图像的实现方式类似，可参照获取第一邻域图像的具体描述，简明起见，此处不再赘述。
S102、根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型和第二点云模型之间的关系，获取合并参数。
可选地，第一点云模型可以是预先根据第一图像集中的各第一图像，并采用SMF算法重建的目标场景的三维点云模型；第二点云模型可以是预先根据第二图像集中的各第二图像，并采用SMF算法重建的目标场景的三维点云模型。当然，也可以采用其他算法根据第一图像集和第二图像集分别重建获得上述第一点云模型和第二点云模型，本方案并不限制。
本方案中，目标图像对可以为一组，也可以为多组。当目标图像对的数量为一组时，可根据获得的一组合并参数指导第一点云模型和第二点云模型的合并。
当目标图像对的数量为多组时，每组目标图像对可以对应目标场景中的不同区域，之后可根据各组目标图像对获得的合并参数有针对性地指导第一点云模型和第二点云模型中相应区域的合并；需要说明的是，在三维场景重建中，目标场景(或者合并之前的点云模型)被划分为多个区域，针对每个区域分别计算获得一组合并参数，合并后的点云模型一致性更强。
示例性地，在重建一栋建筑物的三维结构时，可将建筑物划分为：正面、背面、第一侧面以及第二侧面。针对建筑物的正面，获取目标图像对1；针对建筑物的背面，获取目标图像对2；针对建筑物的第一侧面，获取目标图像对3；针对建筑物的第二侧面，获取目标图像对4。针对目标图像对1至4分别进行计算，获得4组合并参数，其中，目标图像对1对应的合并参数用于合并建筑物的正面的点云，目标图像对2对应的合并参数用于合并建筑物的背面区域的点云，目标图像对3对应的合并参数用于合并建筑物的第一侧面的点云，目标图像对4对应的合并参数用于合并建筑物的第二侧面的点云。
针对每组目标图像对获取对应的合并参数，可通过下述任一种方式：
一种可能的实现方式，可采用自动的方式获取合并参数，具体可以包括以下步骤：
步骤一，根据基于深度学习的特征匹配的方法，获取第一匹配关系；并根据第一匹配关系包括的像素点，获得第二匹配关系以及第三匹配关系。其中，第一匹配关系即目标第一图像和目标第二图像之间的像素点匹配关系，第二匹配关系即目标第一图像与第一邻域图像之间的像素点匹配关系，第三匹配关系即目标第二图像与第二邻域图像之间的像素点匹配关系。
其中，基于深度学习的特征匹配的方法包括：采用深度学习的方法预先训练满足要求的图像匹配模型，利用图像匹配模型获取图像之间的像素点匹配结果。一种可能的实现方式中，图像匹配模型首先对其中一幅图像(假设为图像A)进行热点区域学习，根据各不 同区域的响应值确定热点区域，其中，响应值越高，表示该区域为热点区域的概率越高，响应值越低，表示该区域为热点区域的概率值越低；利用图像匹配模型提取热点区域的像素点的特征信息，该特征信息包括图像纹理信息和图像语义信息；接着，图像匹配模型根据图像A的热点区域的像素点的图像信息和纹理信息，在其他图像中进行匹配，从而获得图像之间的像素点匹配关系。
下面示例性地说明图像之间的像素点匹配关系：
假设A1、A2……An为目标第一图像中的像素点，B1、B2……Bn为目标第二图像中的像素点，第一匹配关系包括A1与B1、A2与B2……An与Bn之间的匹配关系。
根据第一匹配关系中包括的像素点A1、A2……An，分别在第一邻域图像包括的各像素点中进行匹配，基于匹配结果生成第二匹配关系，第二匹配关系包括A1与C1、A2与C2……Am与Cm之间的匹配关系，其中，C1、C2……Cm为第一邻域图像中的像素点。
根据第一匹配关系中包括的像素点B1、B2……Bn，分别在第二邻域图像中进行匹配，基于匹配结果生成第三匹配关系，第三匹配关系包括B1与D1、B2与D2……Bi与Di之间的匹配关系，其中，D1、D2……Di为第二邻域图像的像素点。n、m、i均为大于或等于1的整数，且m小于或等于n，i小于或等于n。
应理解，上述第一匹配关系、第二匹配关系以及第三匹配关系均为二维(2D)的像素点之间的匹配关系。且第二匹配关系和第三匹配关系的数量均可以为多个，第二匹配关系的数量与第一邻域图像的数量相同，第三匹配关系的数量与第二邻域图像的数量相同。
步骤二，根据第二匹配关系以及各第一邻域图像的相机位置，进行特征三角化处理，获得第三点云模型；根据第三匹配关系以及各第二邻域图像的相机位置，进行特征三角化处理，获得第四点云模型。其中，第三点云模型与第一点云模型的坐标系相同且尺度相同；第四点云模型与第二点云模型的坐标系相同且尺度相同。
参照上述示例可知，本方案中，第二匹配关系包括的目标第一图像中的像素点为第一匹配关系包括的目标第一图像的像素点的子集，因此，根据第二匹配关系获得的第三点云模型相当于第一点云模型的局部；类似地，第三匹配关系包括的目标第二图像中的像素点为第一匹配关系包括的目标第二图像的像素点的子集，因此，根据第三匹配关系获得的第四点云模型相当于第二点云模型的局部。针对每个目标场景中的每个局部区域获取差异化的合并参数，能够提高合并后的模型的一致性。
另外，由于第三点云模型与第一点云模型的坐标系和尺度相同，第四点云模型与第二点云模型的坐标系和尺度相同，因此，根据第三点云模型和第四点云模型获取的合并参数能够较好地指导第一点云模型和第二点云模型的合并。
步骤三，根据第三点云模型包括的点云以及第四点云模型包括的点云之间的第四匹配 关系，获取合并参数。
具体地，根据第一匹配关系、第二匹配关系、第三匹配关系、第五匹配关系以及第六匹配关系，获取第四匹配关系，其中，第四匹配关系为第三点云模型包括的点云与第四点云模型包括的点云之间的匹配关系，可以理解为一种3D-3D的对应关系。其中，第五匹配关系为第一邻域图像中的像素点与第三点云模型中的点云之间的匹配关系，第六匹配关系为第二邻域图像中的像素点与第四点云模型中的点云之间的匹配关系。根据第四匹配关系包含的各点云在相应坐标系中的位置，采用基于随机采样一致性的迭代最近邻算法(RANSAC-ICP)，获取所述合并参数。
其中，第一图像集中各第一图像分别与第一点云模型之间的匹配关系在创建第一点云模型时获取。第二图像集中各第二图像分别与第二点云模型之间的匹配关系在创建第二点云模型时获取。
由于第三点云模型和第四点云模型之间的第四匹配关系中可能会存在部分点云误匹配，因此，本方案采用RANSAC-ICP算法去除误匹配，保证正确的匹配。其中，误匹配的点云称为外点，正确匹配的点云称为内点。具体地，采用RANSAC-ICP算法去除误匹配时，首先，随机采样M对(例如，M＝4)具有匹配关系的点云，基于SVD分解的方法，计算获得一组合并参数，记为[s，r，t]，s表示缩放尺度、r表示旋转参数，t表示平移参数；接着，根据合并参数[s，r，t]统计上述第四匹配关系中符合该合并参数的点云对的数量并记录内点数量；重复执行上述步骤W次，取内点数数量最多时对应的合并参数作为最后结果输出。其中，M为大于或等于3的正整数，W为大于或等于1的正整数。
另一种可能的实现方式，可采用人工的方式，需要说明的是，这里所指的“人工的方式”是表示其中部分步骤获取的信息通过人工的方式实现，并不是所有的步骤均通过人工的方式实现。例如，在下述所示的实现方式中，步骤一中获取的目标像素点、目标点云和目标场景中的特征点三者之间的匹配关系是基于人工标注获取的。
示例性地，可以包括以下步骤：
步骤一、获取预先标注的目标场景的特征点分别在目标图像对、第一邻域图像以及第二邻域图像中对应的目标像素点，并获取上述目标场景的特征点分别在第一点云模型和第二点云模型中对应的目标点云。
其中，上述预先标注的目标像素点和目标点云，可以是人工根据目标场景的特征点标注的。应理解，上述目标像素点、目标点云和目标场景中的特征点三者之间具有匹配关系，该匹配关系可以理解为一种2D-3D的对应关系。
步骤二、根据上述目标像素点和目标特征点，获取合并参数。
一种可能的实现方式，根据上述目标像素点分别在所属图像中的位置、目标点云分别 在所属点云模型中的位置，采用基于随机采样一致性的PnP算法(RANSAC-PnP)，获取合并参数。
具体地，由于上述2D-3D的对应关系是人工预先标注的，可能会存在部分2D-3D的对应关系为误匹配，因此，本方案采用RANSAC-PnP去除误匹配，保证正确的匹配。其中，误匹配的点云(或像素点)称为外点，正确匹配的点云(或像素点)称为内点。具体地，采用RANSAC-PnP算法去除误匹配时，首先，随机采样M＇对(例如，M＇＝4)具有匹配关系的目标像素点与目标点云，基于EPnP算法，计算获得一组合并参数，记为[s，r，t]，s表示缩放尺度、r表示旋转参数，t表示平移参数；接着，根据合并参数[s，r，t]统计上述2D-3D的对应关系中符合该合并参数的目标像素点与目标点云对的数量并记录内点数量；重复执行上述步骤W＇次，取内点数数量最多时对应的合并参数作为最后结果输出。其中，W＇为大于或等于1的正整数。
S103、根据所述合并参数，对第一点云模型和第二点云模型进行合并，获取目标点云模型。
具体地，对于每组目标图像对来说，根据该组目标图像对对应的合并参数，对第一点云模型或第二点云模型中相应区域的点云进行合并，例如，缩放处理、旋转处理以及平移处理一项或多项。
本实施例提供的点云模型处理方法，通过从针对目标场景拍摄的，且具有不同视觉效果的第一图像集和第二图像集中，获取至少一组目标图像对以及目标图像对对应的第一邻域图像和第二邻域图像；针对每组目标图像对，根据所述目标图像对、第一邻域图像以及第二邻域图像、第一点云模型和第二点云模型之间的关系，计算获得合并参数；根据各组目标图像对对应的合并参数，对第一点云模型和第二点云模型进行合并，获取用于重建目标场景的三维结构的目标点云模型。本实施例通过使用更多的原始图像信息参与点云模型的合并，相对于现有技术中直接根据点云模型进行合并的方式，本方案获得的合并后的点云模型的一致性更高。
图2为本公开另一实施例提供的点云模型处理方法的流程图。参照图2所示，在图1所示实施例的基础上，S103之后，还可以包括：
S104、根据所述合并参数，将第一图像集中各第一图像的相机位置以及第二图像集中各第二图像的相机位置合并至所述目标点云模型的坐标系中。
具体地，首先，将第一图像集中各第一图像的相机位置先添加至第一点云模型的坐标系中，并将第二图像集中各第二图像的相机位置添加至第二点云模型的坐标系中。接着，根据每组目标图像对的合并参数，对第一点云模型的坐标系中相应区域的各相机位置进行缩放处理、旋转处理以及平移处理中的一项或多项；或者，根据每组目标图像对的合并参 数，对第二点云模型的坐标系中相应区域的各相机位置进行缩放处理、旋转处理以及平移处理中的一项或多项。
本实施例通过利用合并参数，将不同图像集中各图像的拍摄位置对齐至目标点云模型的坐标系中，为后续其他图像处理提供依据。
在图2所示实施例的基础上，可选地，还可以包括步骤S105、S106：
S105、根据集束调整优化算法，对目标点云模型的坐标系中各第一图像的相机位置、各第二图像的新的相机位置以及各点云的位置进行最大似然估计，获取估计结果。
S106、根据估计结果，对所述目标点云模型的坐标系中各第一图像的相机位置、各第二图像的相机位置以及各点云的位置分别进行调整。
由于获取第一图像集时，相机位置误差会累积，进而会对第三点云模型的精度产生影响；类似地，获取第二图像集时，相机位置误差累积，进而会对第四点云模型的精度产生影响。因此，本实施例通过集束调整优化算法进行全局优化，减小三维点到二维图像点的重投影误差，使调整后的目标点云模型一致性更强。
可选地，在图1或图2所示实施例的基础上，还可以包括：S107、根据所述目标点云模型重建所述目标场景的三维结构。
可以理解，本方案获得的目标点云模型更为精准，因而恢复的目标场景的三维结构也更为精准。需要说明的是，该步骤未在图1以及图2中体现。
图3A-图3G为本公开另一实施例提供的点云模型处理方法的流程图。在图3A-图3G所示的具体实施例，以第一图像集包括针对目标场景在白天拍摄的图像的集合、第二图像集包括针对目标场景在夜晚拍摄的图像的集合为例进行说明。
参照图3A所示，第一图像集以白天图像数据库为示例，白天图像数据库包括针对目标场景在白天拍摄的多个图像，白天图像数据库中的图像可以称为白天图像样本；第二图像集以夜晚图像数据库为示例，夜晚图像数据库包括针对目标场景在夜晚拍摄的多个图像，夜晚图像数据库中的图像可以称为夜晚图像样本；预先根据白天图像数据库重建的点云模型为三维点云模型A，预先根据夜晚图像数据库重建的点云模型为三维点云模型B。
结合图3A以及图3B所示，通过在图3A所示的白天图像数据库和夜晚图像数据库中进行自动或人工的图像检索，获取目标图像对，以及目标图像对的第一邻域图像和第二邻域图像。
其中，图3B示出了一组目标图像对的情况，参照图3B所示，白天图像样本R1和夜晚图像样本S1为一组目标图像对，白天图像样本R2、R3以及R4均为R1的第一邻域图像，夜晚图像样本S2、S3以及S4均为S1的第二邻域图像。
针对R1和S1，采用基于深度学习算法或人工标注的方法，获取R1与S1之间的像素 点匹配关系。参照图3C所示，虚线段两端的箭头分别连接的实心六角星和实心圆圈为一组匹配的像素点，其中，实心六角星表示白天图像样本R1中具有匹配关系的像素点，实心圆圈表示夜晚图像样本S1中具有匹配关系的像素点。
需要说明的是，在图3C所示的情况中，白天图像样本R1和夜晚图像样本S1中不具备匹配关系的像素点并未通过标记示出。
在接下来的处理中，若采用自动的方式，请参照图3D所示；若采用人工的方式，请参照图3E所示。
自动的方式：
参照图3D所示，首先根据白天图像样本R1中具有匹配关系的像素点在白天图像样本R2、R3以及R4中进行匹配；并根据夜晚图像样本S1中具有匹配关系的像素点在夜晚图像样本S2、S3以及S4中进行匹配；基于白天图像样本R1分别与白天图像样本R2、R3以及R4的匹配结果进行特征三角化处理，获得三维点云模型A＇；基于夜晚图像样本S1分别与夜晚图像样本S2、S3以及S4的匹配结果进行特征三角化处理，获得三维点云模型B＇。
接着，利用白天图像样本R1分别与白天图像样本R2、R3以及R4的匹配结果、夜晚图像样本S1分别与夜晚图像样本S2、S3以及S4的匹配结果、以及白天图像样本R1和夜晚图像样本S1之间的匹配关系，将三维点云模型A＇和三维点云模型B＇中的点云进行关联，从而获取了3D-3D的对应关系。
参照图3D所示，空心六角星表示三维点云模型A＇中的点云，空心圆圈表示三维点云模型B＇中的点云，虚线段两端的箭头连接的空心六角形和空心圆圈为一组具有匹配关系的点云。
人工的方式：
参照图3E所示，可将人工选取的白天图像样本R1、R2、R3以及R4中的特征点与三维点云模型A和/或三维点云模型B中对应的点云进行关联；并将人工选取的夜晚图像样本S1、S2、S3以及S4中的特征点与三维点云模型A和/或三维点云模型B中对应的点云进行关联，从而获取了2D-3D的对应关系。
其中，人工在白天图像样本中选取的特征点和人工在夜晚图像样本中选取的特征点通常指向目标场景中的同一特征点。指向目标场景中的同一特征点可从三维点云模型A或者三维点云模型B中恢复了三维结构的点来确定。
参照图3E所示，空心六角星表示三维点云模型A中的点云，空心圆圈表示三维点云模型B中的点云，虚线段两端的箭头连接的空心六角形和空心圆圈为一组具有匹配关系的点云，虚线段两端的箭头连接的空心圆圈和实心圆圈为一组具有匹配关系的点云和像素点， 虚线段两端的箭头连接的实心六角形和实心圆圈为一组具有匹配关系的像素点，虚线段两端的箭头连接的空心六角形和实心六角形为一组具有匹配关系的点云和像素点。
且图3E仅示出了部分像素点与点云之间的匹配关系。
利用RANSAC-ICP对图3D所示的方式获取的3D-3D的对应关系进行计算，或者，利用RANSAC-PnP对图3E所示的方式获取的2D-3D的对应关系进行计算，获取三维点云模型A和三维点云模型B之间的合并参数，合并参数表示为[s，r，t]，其中，s表示缩放尺度、r表示旋转参数、t表示平移参数；并根据合并参数[s，r，t]合并点三维云模型A和三维点云模型B。
参照图3F所示，利用求解出来的合并参数[s，r，t]对三维点云模型A的各点云进行坐标系转换，并将转换后的三维点云模型A的各点云与三维点云模型B的各点云进行相应的合并操作，从而将点云模型A对齐至点云模型B中，并合并三维点云模型A和三维点云模型B中点云，得到的新的点云模型即为前述实施例中所述的目标点云模型。
图3F中是对三维点云模型A进行坐标系转换，实际应用中，也可以根据合并参数，对三维点云模型B进行坐标系转换，将转换后的三维点云模型B合并至三维点云模型A的坐标系中。
在实际应用中，还可以将白天图像数据库中各白天图像样本的相机位置根据合并参数，合并至三维点云模型B的坐标系，并将夜晚图像数据库中各夜晚图像样本的相机位置合并至三维点云模型B的坐标系。
在实际应用中，可将上述获得的3D-3D的对应关系或者2D-3D的对应关系合并至目标点云模型中。参照图3G所示，空心四角星表示合并后目标点云模型中的点云，虚线段两端的箭头连接的空心四角星和实心六角星表示具有匹配关系的点云和像素点，虚线段两端的箭头连接的空心四角星和实心圆圈表示具有匹配关系的点云和像素点。图3G所示的线段即体现了上述3D-3D的对应关系或者2D-3D的对应关系。
需要说明的是，为了减小重建后的三维点到二维图像点的重投影误差，提高目标点云模型的一致性，还可以根据图3F得到的目标点云模型中，各白天图像样本的相机位置、各夜晚图像样本的相机位置、以及各点云的位置，采用集束调整优化算法对图3F中得到的目标点云模型进行全局优化。
以一栋建筑物为例，图4为采用本公开实施例提供的点云模型处理方法进行全局优化前和进行全局优化后分别对应的合并效果示意图，其中，图4中(a)为采用本公开实施例提供的点云模型处理方法，但未进行全局优化的合并效果示意图，图4中(b)为采用本公开实施例提供的点云模型处理方法，且进行了全局优化后的合并效果示意图。对比图4中(a)和(b)分别所示的两种情况，参照图4中(b)所示，在靠近建筑物的两侧处包含很 多通过集束调整优化算法获得的新增的点云，且图4中(b)所示的点云模型的位置也根据估计结果进行了微调，使得点云模型更为精准，因此，可知采用集束调整优化算法进行全局优化后的目标点云模型中包含更多新增的点云，且优化后的点云模型的位置也更为精准，能够得到的点云模型一致性更强。
图5为本公开一实施例提供的点云模型处理装置的结构示意图。参照图5所示，本实施例的提供的点云模型处理装置500，包括：图像提取模块501、参数计算模块502、合并模块503。
其中，图像提取模块501，用于根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及所述目标图像对对应的第一邻域图像和第二邻域图像；其中，所述第一图像集包括具有第一视觉效果的多幅第一图像；所述第二图像集包括具有第二视觉效果的多幅第二图像；所述目标图像对包括相似度满足第一条件的目标第一图像和目标第二图像；所述第一邻域图像包括所述第一图像集中与所述目标第一图像的相似度满足第二条件的图像；所述第二邻域图像包括所述第二图像集中与所述目标第二图像的相似度满足第三条件的图像。
参数计算模块502，用于根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型和第二点云模型之间的关系，获取合并参数；其中，所述第一点云模型是预先根据所述第一图像集重建的三维点云模型；所述第二点云模型是预先根据所述第二图像集重建的三维点云模型。
合并模块503，用于根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型。
在一些可能的设计中，图像提取模块501，具体用于将所述第一图像集和所述第二图像集输入至预先训练好的图像检索模型中，获取所述图像检索模型输出的目标图像对、以及所述目标图像对对应的第一邻域图像和第二邻域图像。
在一些可能的设计中，参数计算模块502，具体用于根据第二匹配关系、所述第一邻域图像对应的相机位置，进行特征三角化处理，获取第三点云模型；其中，所述第二匹配关系包括所述目标第一图像与所述第一邻域图像两者之间的像素点匹配关系，所述第三点云模型与所述第一点云模型的坐标系相同且尺度相同；根据第三匹配关系及所述第二邻域图像对应的相机位置，进行特征三角化处理，获取第四点云模型；其中，所述第三匹配关系包括所述目标第二图像与所述第二邻域图像两者之间的像素点匹配关系，所述第四点云模型与所述第二点云模型的坐标系相同且尺度相同；根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的第四匹配关系，获取所述合并参数。
在一些可能的设计中，所述第二匹配关系和所述第三匹配关系均是根据第一匹配关系 获得的，其中，所述第一匹配关系为所述目标图像对包括的目标第一图像和目标第二图像之间的像素点匹配关系。
在一些可能的设计中，参数计算模块502，具体用于根据第一匹配关系、第二匹配关系、第三匹配关系、第五匹配关系以及第六匹配关系，获取所述第四匹配关系，其中，所述第五匹配关系为所述第一邻域图像中的像素点与所述第三点云模型中的点云之间的匹配关系，所述第六匹配关系为所述第二邻域图像中的像素点与所述第四点云模型中的点云之间的匹配关系；根据所述第四匹配关系包含的各点云在相应坐标系中的位置，获取所述合并参数。
在一些可能的设计中，参数计算模块502，具体用于根据所述第四匹配关系以及所述第四匹配关系包含的各点云在相应坐标系中的位置，采用基于随机采样一致性的迭代最近邻算法，获取所述合并参数。
在一些可能的设计中，合并模块503，还用于根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型之后，将所述第四匹配关系，合并至所述目标点云模型中。
在一些可能的设计中，参数计算模块502，具体用于获取预先标注的所述目标场景中的特征点分别在所述目标图像对、所述第一邻域图像以及所述第二邻域图像中对应的目标像素点、及预先标注的所述特征点分别在所述第一点云模型和所述第二点云模型中对应的点云；根据所述特征点对应的目标像素点及目标点云，获取所述合并参数。
在一些可能的设计中，参数计算模块502，具体用于根据各所述目标像素点在所属图像中的位置，及所述目标点云在所属点云模型中的位置，采用基于随机采样一致性的PnP算法，获取所述合并参数。
在一些可能的设计中，合并模块503，还用于所述根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型之后，将所述特征点、所述目标像素点以及所述目标点云三者之间的对应关系，合并至所述目标点云模型中。
在一些可能的设计中，合并模块503，还用于根据所述合并参数，将所述第一图像集中各第一图像的相机位置以及所述第二图像集中各第二图像的相机位置合并至所述目标点云模型中。
在一些可能的设计中，点云模型处理装置500，还包括：点云模型优化模块504。其中，点云模型优化模块504，用于根据集束调整优化算法，对所述目标点云模型中各所述第一图像的相机位置、各所述第二图像的新的相机位置以及各点云的位置进行最大似然估计，获取估计结果；根据所述估计结果，对所述目标点云模型中各所述第一图像的相机位置、各所述第二图像的相机位置以及各点云的位置分别进行调整。
本实施例提供的点云模型处理装置能够用于执行前述任一方法实施例的技术方案，其实现原理以及技术效果类似，可参照前述实施例的描述，此处不再赘述。
图6为本公开一实施例提供的电子设备的结构示意图。参照图6所示，本实施例提供的电子设备600，包括：存储器601和处理器602。
其中，存储器601可以是独立的物理单元，与处理器602可以通过总线603连接。存储器601、处理器602也可以集成在一起，通过硬件实现等。
存储器601用于存储程序指令，处理器602调用该程序指令，执行以上任一方法实施例的操作。
可选地，当上述实施例的方法中的部分或全部通过软件实现时，上述电子设备600也可以只包括处理器602。用于存储程序的存储器601位于电子设备600之外，处理器602通过电路/电线与存储器连接，用于读取并执行存储器中存储的程序。
处理器602可以是中央处理器(central processing unit，CPU)，网络处理器(network processor，NP)或者CPU和NP的组合。
处理器602还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合。
存储器601可以包括易失性存储器(volatile memory)，例如随机存取存储器(random-access memory，RAM)；存储器也可以包括非易失性存储器(non-volatile memory)，例如快闪存储器(flash memory)，硬盘(hard disk drive，HDD)或固态硬盘(solid-state drive，SSD)；存储器还可以包括上述种类的存储器的组合。
本公开还提供一种计算机可读存储介质，计算机可读存储介质中包括计算机程序指令，所述计算机程序指令在被电子设备的至少一个处理器执行时，执行以上任一方法实施例的技术方案。
本公开还提供一种程序产品，所述程序产品包括计算机程序，所述计算机程序存储在可读存储介质中，所述电子设备的至少一个处理器可以从所述可读存储介质中读取所述计算机程序，所述至少一个处理器执行所述计算机程序使得所述电子设备执行如上任一方法实施例的技术方案。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体 意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本公开的具体实施方式，使本领域技术人员能够理解或实现本公开。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本公开的精神或范围的情况下，在其它实施例中实现。因此，本公开将不会被限制于本文所述的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (16)

  1. 一种点云模型处理方法，其特征在于，包括：
    根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及所述目标图像对对应的第一邻域图像和第二邻域图像；其中，所述第一图像集包括具有第一视觉效果的多幅第一图像；所述第二图像集包括具有第二视觉效果的多幅第二图像；所述目标图像对包括相似度满足第一条件的目标第一图像和目标第二图像；所述第一邻域图像包括所述第一图像集中与所述目标第一图像的相似度满足第二条件的图像；所述第二邻域图像包括所述第二图像集中与所述目标第二图像的相似度满足第三条件的图像；
    根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型、以及第二点云模型之间的关系，获取合并参数；其中，所述第一点云模型包括预先根据所述第一图像集重建的三维点云模型；所述第二点云模型包括预先根据所述第二图像集重建的三维点云模型；
    根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型。
  2. 根据权利要求1所述的方法，其特征在于，所述根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及述目标图像对对应的第一邻域图像和第二邻域图像，包括：
    将所述第一图像集和所述第二图像集输入至预先训练好的图像检索模型中，获取所述图像检索模型输出的目标图像对、以及所述目标图像对对应的第一邻域图像和第二邻域图像。
  3. 根据权利要求1或2所述的方法，其特征在于，所述根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型、和第二点云模型之间的关系，获取合并参数，包括：
    根据第二匹配关系、所述第一邻域图像对应的相机位置，进行特征三角化处理，获取第三点云模型；其中，所述第二匹配关系包括所述目标第一图像与所述第一邻域图像两者之间的像素点匹配关系，所述第三点云模型与所述第一点云模型的坐标系相同且尺度相同；
    根据第三匹配关系及所述第二邻域图像对应的相机位置，进行特征三角化处理，获取第四点云模型；其中，所述第三匹配关系包括所述目标第二图像与所述第二邻域图像两者之间的像素点匹配关系，所述第四点云模型与所述第二点云模型的坐标系相同且尺度相同；
    根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的第四匹配关系，获取所述合并参数。
  4. 根据权利要求3所述的方法，其特征在于，所述第二匹配关系和所述第三匹配关系均是根据第一匹配关系获得的，其中，所述第一匹配关系为所述目标图像对包括的目标第 一图像和目标第二图像之间的像素点匹配关系。
  5. 根据权利要求4所述的方法，其特征在于，所述根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的点云匹配关系，获取所述合并参数，包括：
    根据第一匹配关系、第二匹配关系、第三匹配关系、第五匹配关系以及第六匹配关系，获取所述第四匹配关系，其中，所述第五匹配关系为所述第一邻域图像中的像素点与所述第三点云模型中的点云之间的匹配关系，所述第六匹配关系为所述第二邻域图像中的像素点与所述第四点云模型中的点云之间的匹配关系；
    根据所述第四匹配关系包含的各点云在相应坐标系中的位置，获取所述合并参数。
  6. 根据权利要求3所述的方法，其特征在于，所述根据所述第三点云模型包括的点云与所述第四点云模型包括的点云之间的第四匹配关系，获取所述合并参数，包括：
    根据所述第四匹配关系以及所述第四匹配关系包含的各点云在相应坐标系中的位置，采用基于随机采样一致性的迭代最近邻算法，获取所述合并参数。
  7. 根据权利要求3所述的方法，其特征在于，所述根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型之后，还包括：
    将所述第四匹配关系，合并至所述目标点云模型中。
  8. 根据权利要求1所述的方法，其特征在于，所述根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型和第二点云模型之间的关系，获取合并参数，包括：
    获取预先标注的所述目标场景中的特征点分别在所述目标图像对、所述第一邻域图像以及所述第二邻域图像中对应的目标像素点、及预先标注的所述特征点分别在所述第一点云模型和所述第二点云模型中对应的点云；
    根据所述特征点对应的目标像素点及目标点云，获取所述合并参数。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述特征点对应的目标像素点及目标点云，获取所述合并参数，包括：
    根据各所述目标像素点在所属图像中的位置，及所述目标点云在所属点云模型中的位置，采用基于随机采样一致性的PnP算法，获取所述合并参数。
  10. 根据权利要求8所述的方法，其特征在于，所述根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型之后，还包括：
    将所述特征点、所述目标像素点以及所述目标点云三者之间的对应关系，合并至所述目标点云模型中。
  11. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    根据所述合并参数，将所述第一图像集中各第一图像的相机位置以及所述第二图像集 中各第二图像的相机位置合并至所述目标点云模型中。
  12. 根据权利要求11所述的方法，其特征在于，所述方法还包括：
    根据集束调整优化算法，对所述目标点云模型中各所述第一图像的相机位置、各所述第二图像的新的相机位置以及各点云的位置进行最大似然估计，获取估计结果；
    根据所述估计结果，对所述目标点云模型中各所述第一图像的相机位置、各所述第二图像的相机位置以及各点云的位置分别进行调整。
  13. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    根据所述目标点云模型重建所述目标场景的三维结构。
  14. 一种点云模型处理装置，其特征在于，包括：
    图像提取模块，用于根据针对目标场景拍摄的第一图像集和第二图像集，获取目标图像对以及所述目标图像对对应的第一邻域图像和第二邻域图像；其中，所述第一图像集包括具有第一视觉效果的多幅第一图像；所述第二图像集包括具有第二视觉效果的多幅第二图像；所述目标图像对包括相似度满足第一条件的目标第一图像和目标第二图像；所述第一邻域图像包括所述第一图像集中与所述目标第一图像的相似度满足第二条件的图像；所述第二邻域图像包括所述第二图像集中与所述目标第二图像的相似度满足第三条件的图像；
    参数计算模块，用于根据所述目标图像对、所述第一邻域图像、所述第二邻域图像、第一点云模型和第二点云模型之间的关系，获取合并参数；其中，所述第一点云模型是预先根据所述第一图像集重建的三维点云模型；所述第二点云模型是预先根据所述第二图像集重建的三维点云模型；
    合并模块，用于根据所述合并参数，对所述第一点云模型和所述第二点云模型进行合并，获取目标点云模型。
  15. 一种电子设备，其特征在于，包括：存储器、处理器以及计算机程序指令；
    所述存储器被配置为存储所述计算机程序指令；
    所述处理器被配置为执行所述计算机程序指令，当所述处理器执行所述计算机程序指令时，实现如权利要求1至13任一项所述的点云模型处理方法。
  16. 一种可读存储介质，其特征在于，包括：程序；
    所述程序被电子设备的至少一个处理器执行时，所述电子设备实现如权利要求1至13任一项所述的点云模型处理方法。
PCT/CN2022/084109 2021-05-13 2022-03-30 点云模型处理方法、装置及可读存储介质 Ceased WO2022237368A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/283,705 US12511826B2 (en) 2021-05-13 2022-03-30 Point cloud model processing method and apparatus, and readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110524280.6 2021-05-13
CN202110524280.6A CN115346016A (zh) 2021-05-13 2021-05-13 点云模型处理方法、装置及可读存储介质

Publications (1)

Publication Number Publication Date
WO2022237368A1 true WO2022237368A1 (zh) 2022-11-17

Family

ID=83946867

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/084109 Ceased WO2022237368A1 (zh) 2021-05-13 2022-03-30 点云模型处理方法、装置及可读存储介质

Country Status (3)

Country Link
US (1) US12511826B2 (zh)
CN (1) CN115346016A (zh)
WO (1) WO2022237368A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115937466A (zh) * 2023-02-17 2023-04-07 烟台市地理信息中心 一种融合gis的三维模型生成方法、系统及存储介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8874454B2 (en) * 2013-03-15 2014-10-28 State Farm Mutual Automobile Insurance Company Systems and methods for assessing a roof
CN117455936B (zh) * 2023-12-25 2024-04-12 法奥意威(苏州)机器人系统有限公司 点云数据处理方法、装置及电子设备
US12614309B2 (en) * 2024-06-28 2026-04-28 Qualcomm Incorporated Online intrinsic calibration
CN119672235B (zh) * 2025-02-19 2025-06-13 江西师范大学 一种基于无人机的带有浮雕的三维建筑生成方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080310757A1 (en) * 2007-06-15 2008-12-18 George Wolberg System and related methods for automatically aligning 2D images of a scene to a 3D model of the scene
US20140218353A1 (en) * 2013-02-01 2014-08-07 Apple Inc. Image group processing and visualization
US20180276885A1 (en) * 2017-03-27 2018-09-27 3Dflow Srl Method for 3D modelling based on structure from motion processing of sparse 2D images
CN108921939A (zh) * 2018-07-04 2018-11-30 王斌 一种基于图片的三维场景重建方法

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9953459B2 (en) * 2008-11-05 2018-04-24 Hover Inc. Computer vision database platform for a three-dimensional mapping system
US10848731B2 (en) * 2012-02-24 2020-11-24 Matterport, Inc. Capturing and aligning panoramic image and depth data
US11282287B2 (en) * 2012-02-24 2022-03-22 Matterport, Inc. Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications
US11080286B2 (en) * 2013-12-02 2021-08-03 Autodesk, Inc. Method and system for merging multiple point cloud scans
US9443134B2 (en) * 2014-05-15 2016-09-13 Adobe Systems Incorporated Propagating object selection across multiple images
US10475234B2 (en) * 2015-07-15 2019-11-12 George Mason University Multi-stage method of generating 3D civil site surveys
US11172824B2 (en) * 2016-04-06 2021-11-16 Carestream Dental Technology Topco Limited Hybrid OCT and surface contour dental imaging
US10317199B2 (en) * 2016-04-08 2019-06-11 Shining 3D Tech Co., Ltd. Three-dimensional measuring system and measuring method with multiple measuring modes
US10650590B1 (en) * 2016-09-07 2020-05-12 Fastvdo Llc Method and system for fully immersive virtual reality
US10198829B2 (en) * 2017-04-25 2019-02-05 Symbol Technologies, Llc Systems and methods for extrinsic calibration of a plurality of sensors
WO2019002616A1 (en) * 2017-06-30 2019-01-03 Carestream Dental Technology Topco Limited SURFACE CARTOGRAPHY USING AN INTRA-MOBILE SCANNER HAVING PENETRATION CAPABILITIES
US11443469B2 (en) * 2017-09-27 2022-09-13 Shutterfly, Llc System and method for reducing similar photos for display and product design
US10762126B2 (en) * 2017-09-27 2020-09-01 Shutterfly, Llc System and method for reducing similar photos for display and product design
US10970425B2 (en) * 2017-12-26 2021-04-06 Seiko Epson Corporation Object detection and tracking
EP3803776A4 (en) * 2018-05-25 2022-03-23 Packsize, LLC SYSTEMS AND METHODS FOR MULTI-CAMERA PLACEMENT
US10699458B2 (en) * 2018-10-15 2020-06-30 Shutterstock, Inc. Image editor for merging images with generative adversarial networks
CN111366938B (zh) * 2018-12-10 2023-03-14 北京图森智途科技有限公司 一种挂车夹角的测量方法、装置及车辆
US11741566B2 (en) * 2019-02-22 2023-08-29 Dexterity, Inc. Multicamera image processing
US11170552B2 (en) * 2019-05-06 2021-11-09 Vangogh Imaging, Inc. Remote visualization of three-dimensional (3D) animation with synchronized voice in real-time
US11232633B2 (en) * 2019-05-06 2022-01-25 Vangogh Imaging, Inc. 3D object capture and object reconstruction using edge cloud computing resources
WO2021062645A1 (en) * 2019-09-30 2021-04-08 Zte Corporation File format for point cloud data
US11074701B2 (en) * 2019-12-13 2021-07-27 Reconstruct Inc. Interior photographic documentation of architectural and industrial environments using 360 panoramic videos
US11335063B2 (en) * 2020-01-03 2022-05-17 Vangogh Imaging, Inc. Multiple maps for 3D object scanning and reconstruction
US12026834B2 (en) * 2021-06-15 2024-07-02 Reconstruct Inc. Method to determine from photographs the placement and progress of building elements in comparison with a building plan
US11741631B2 (en) * 2021-07-15 2023-08-29 Vilnius Gediminas Technical University Real-time alignment of multiple point clouds to video capture
US12062245B2 (en) * 2021-08-31 2024-08-13 Vilnius Gediminas Technical University System and method for real-time creation and execution of a human digital twin
US11941827B2 (en) * 2021-10-19 2024-03-26 Datalogic Ip Tech S.R.L. System and method of 3D point cloud registration with multiple 2D images

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080310757A1 (en) * 2007-06-15 2008-12-18 George Wolberg System and related methods for automatically aligning 2D images of a scene to a 3D model of the scene
US20140218353A1 (en) * 2013-02-01 2014-08-07 Apple Inc. Image group processing and visualization
US20180276885A1 (en) * 2017-03-27 2018-09-27 3Dflow Srl Method for 3D modelling based on structure from motion processing of sparse 2D images
CN108921939A (zh) * 2018-07-04 2018-11-30 王斌 一种基于图片的三维场景重建方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
TOLDO, R. ET AL.: "Hierarchical structure-and-motion recovery from uncalibrated images", COMPUTER VISION AND IMAGE UNDERSTANDING, vol. 140, 6 June 2015 (2015-06-06), XP029269234, DOI: 10.1016/j.cviu.2015.05.011 *
WU ZHENGZHENG, KOU ZHAN: "3D Reconstruction of Scene Based on Monocular Multi-View Image", OPTICS & OPTOELECTRONIC TECHNOLOGY, vol. 18, no. 5, 10 October 2020 (2020-10-10), pages 51 - 56, XP093004190, ISSN: 1672-3392, DOI: 10.19519/j.cnki.1672-3392.2020.05.010 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115937466A (zh) * 2023-02-17 2023-04-07 烟台市地理信息中心 一种融合gis的三维模型生成方法、系统及存储介质

Also Published As

Publication number Publication date
CN115346016A (zh) 2022-11-15
US20240161392A1 (en) 2024-05-16
US12511826B2 (en) 2025-12-30

Similar Documents

Publication Publication Date Title
WO2022237368A1 (zh) 点云模型处理方法、装置及可读存储介质
CN111815757B (zh) 基于图像序列的大型构件三维重建方法
CN111383333B (zh) 一种分段式sfm三维重建方法
Sheng et al. Unsupervised collaborative learning of keyframe detection and visual odometry towards monocular deep slam
CN114140527B (zh) 一种基于语义分割的动态环境双目视觉slam方法
CN107292949B (zh) 场景的三维重建方法、装置及终端设备
CN111627065A (zh) 一种视觉定位方法及装置、存储介质
CN109410316B (zh) 物体的三维重建的方法、跟踪方法、相关装置及存储介质
Micusik et al. Descriptor free visual indoor localization with line segments
CN112734839B (zh) 一种提高鲁棒性的单目视觉slam初始化方法
CN112634305B (zh) 一种基于边缘特征匹配的红外视觉里程计实现方法
CN115049782B (zh) 重建稠密三维模型的方法、装置及可读存储介质
Shi et al. Dense semantic 3D map based long-term visual localization with hybrid features
CN117540043B (zh) 基于跨实例和类别对比的三维模型检索方法及系统
CN119784583A (zh) 一种自适应全景图像无缝拼接方法
CN118968514A (zh) 一种物体级语义重定位方法、装置、终端及存储介质
CN115937002B (zh) 用于估算视频旋转的方法、装置、电子设备和存储介质
CN116862969A (zh) 一种双目视差估计方法及其应用
CN119027496A (zh) 一种基于点线特征的单目视觉定位方法、芯片及机器人
CN116229577A (zh) 基于rgbd多模态信息的三维人体位姿估计方法及装置
Zhang et al. Reconstructing 3D Scenes from UAV Images Using a Structure-from-Motion Pipeline
WO2013173383A1 (en) Methods and apparatus for processing image streams
Lang et al. Iftd: Image feature triangle descriptor for loop detection in driving scenes
CN115690137B (zh) 一种终端、图像的光流追踪方法、装置、芯片和存储介质
TWI823491B (zh) 深度估計模型的優化方法、裝置、電子設備及存儲介質

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22806338

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18283705

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22806338

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 18283705

Country of ref document: US