JP2023540917A - 3D reconstruction and related interactions, measurement methods and related devices and equipment - Google Patents

Abstract

The present disclosure provides three-dimensional reconstruction and related interactions, measurement methods and related apparatus and devices. The 3D reconstruction method involves acquiring multiple frames of images to be processed obtained by scanning a target to be reconstructed with a shooting device, and using the images to be processed for each frame and the calibration parameters of the shooting device. to determine the target pixel point belonging to the reconstruction target of the image to be processed in each frame and its camera pose parameter; and to sequentially divide the image data of the image to be processed in each frame into corresponding data sets. , determining pose optimization parameters for the dataset using the image data of the dataset, and the image data and pose optimization parameters of the dataset before which the time series is placed. The above solution can enhance the effect of three-dimensional reconstruction and reduce the computational load of three-dimensional reconstruction.

Description

(Cross reference to related applications)
The present disclosure relates to a Chinese patent application with the application number 202110031502.0, the filing date of January 11, 2021, and the application title of "Three-dimensional reconstruction and related interaction, measurement method and related devices and equipment" claims priority to the Chinese patent application, the entire contents of which are hereby incorporated by reference into this disclosure.

The present disclosure relates to the field of computer vision technology, and particularly to three-dimensional reconstruction and related interactions, measurement methods, and related devices and equipment.

With the development of information technology and the improvement of the level of electronic technology, people use mobile terminals with integrated photographic devices, such as mobile phones, tablet computers, etc., to reconstruct objects in real scenes in three dimensions. Therefore, the 3D model obtained by 3D reconstruction is used to realize applications such as Augmented Reality (AR), games, etc. on mobile terminals.

However, when the photographing device scans and photographs objects in the real scene, the images obtained from the photographing device have different degrees of noise, and the current mainstream camera pose parameter solving method will result in some errors. Inevitably, errors continue to accumulate as scanning and imaging progresses, affecting the effectiveness of the 3D model, and throughout the 3D reconstruction process, as the field of view of scanning and imaging increases, new Captured images are continuously incorporated, and the computational load gradually increases. In view of this, how to enhance the effect of 3D reconstruction and reduce the computational load for 3D reconstruction becomes an urgent problem to be solved.

The present disclosure provides a three-dimensional reconstruction method and related devices and equipment.

A three-dimensional reconstruction method according to a first aspect of the present disclosure includes the steps of: acquiring a plurality of frames of images to be processed obtained by an imaging device scanning a target to be reconstructed; Determining the target pixel point belonging to the reconstruction target of the image to be processed in each frame using the calibration parameters of the imaging device and its camera pose parameters, and processing each frame according to a predetermined segmentation policy. Sequentially dividing the image data of a waiting image into corresponding data sets, the image data including at least a target pixel point, the image data of each data set, and the data whose time series is arranged before it. The image data and pose optimization parameters of the datasets are used sequentially to determine the pose optimization parameters of each dataset, and the pose optimization parameters of each dataset are used to determine the pose optimization parameters of the datasets. Performing reconstruction processing on the image data of the image to be processed by adjusting the camera pose parameters of the image to be processed to which it belongs, and using a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the image to be processed. and obtaining a three-dimensional model of the target awaiting reconstruction.

Therefore, the target pixel point belonging to the reconstruction target of the image to be processed of each frame and its camera pose parameter and sequentially divides the image data of each frame's pending image into corresponding datasets according to a predetermined division policy, thereby dividing the image data of each dataset and the data whose time series is placed before it. The image data and pose optimization parameters of the set are used sequentially to determine the pose optimization parameters for each dataset, and each dataset's pose optimization parameters are determined based on the pose parameters of the preceding dataset. Therefore, if the pose optimization parameters of each dataset are used to adjust the camera pose parameters of the pending images to which the image data contained in the dataset belongs, the camera pose parameters accumulated during the scanning process To help eliminate errors in the process, the reconstruction process is performed on the image data of the image to be processed using a predetermined 3D reconstruction method and the adjusted camera pose parameters of the image to be processed. By effectively increasing the effectiveness of the 3D model of the target that is waiting for reconstruction, and by eliminating errors in camera pose parameters for each dataset, the amount of calculation can be reduced, thereby reducing the calculation load. useful for.

An interaction method based on three-dimensional reconstruction according to a second aspect of the present disclosure is to obtain a three-dimensional model of a target awaiting reconstruction, the three-dimensional model being obtained by the three-dimensional reconstruction method according to the first aspect. and constructing a three-dimensional map of the scene where the photographing device is placed by a predetermined visual inertial navigation method, obtaining current pose information of the photographing device in the three-dimensional map, and based on the pose information. , displaying the three-dimensional model on a scene image currently being captured by the imaging device.

Therefore, by displaying the 3D model of the target to be reconstructed on the currently captured scene image based on the pose information of the imaging device in the 3D map of the scene, the geometry of the virtual object and the real scene can be Since the 3D model can be obtained by the 3D reconstruction method in the first aspect, the effect of 3D reconstruction can be improved, and the geometry of virtual and reality can be improved. The fusion effect of virtual consistency can be enhanced, which helps improve user experience.

A measurement method based on three-dimensional reconstruction according to a third aspect of the present disclosure is to obtain a three-dimensional model of a target awaiting reconstruction, and the three-dimensional model is obtained by the three-dimensional reconstruction method according to the first aspect. What is obtained is that the user receives multiple measurement points set in the 3D model, obtains the distances between the multiple measurement points, and calculates the positions corresponding to the multiple measurement points on the target awaiting reconstruction. and obtaining the distance between.

Therefore, by receiving multiple measurement points set by the user in the 3D model, the distances between the multiple measurement points can be obtained, and the distances between the positions corresponding to the multiple measurement points on the target awaiting reconstruction can be obtained. The distance can be obtained, thereby meeting the measurement needs of objects in real scenes, and the 3D model can be obtained by the 3D reconstruction method in the above first aspect, so the effect of 3D reconstruction can be improved. It is possible to further improve measurement accuracy.

A three-dimensional reconstruction device according to a fourth aspect of the present disclosure includes an image acquisition module, a first determination module, a data division module, a second determination module, a parameter adjustment module, and a model reconstruction module, and the image acquisition module includes: The imaging device is configured to obtain a plurality of frames of pending images obtained by scanning the reconstruction target, and the first determination module is configured to obtain a plurality of frames of pending images and calibration parameters of the imaging device. is configured to determine the target pixel point belonging to the reconstruction target of the pending image of each frame and its camera pose parameters using The second determination module is configured to sequentially divide the image data of the image to be processed into corresponding datasets, the image data including at least the target pixel point, and the second determination module is configured to sequentially divide the image data of the image to be processed into corresponding datasets, the image data including at least the target pixel point, The parameter adjustment module is configured to determine pose optimization parameters for each dataset using sequentially the image data and pose optimization parameters of the datasets located in the The model reconstruction module is configured to use a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the pending image to which the image data included in the dataset belongs. The camera pose parameter is configured to perform a reconstruction process on the image data of the image to be processed to obtain a three-dimensional model of the target to be reconstructed.

A three-dimensional reconstruction-based interaction device according to a fifth aspect of the present disclosure includes a model acquisition module, a mapping and positioning module, and a display interaction module, the model acquisition module configured to acquire a three-dimensional model of a target awaiting reconstruction. configured, a three-dimensional model is obtained by the three-dimensional reconstruction method according to the fourth aspect, and the mapping and positioning module constructs a three-dimensional map of the scene in which the imaging device is located according to a predetermined visual-inertial navigation method. , the display interaction module is configured to obtain current pose information of the photographing device in the three-dimensional map, and the display interaction module is configured to display the three-dimensional model in the scene image currently being photographed by the photographing device based on the pose information. configured.

A measurement device based on three-dimensional reconstruction according to a sixth aspect of the present disclosure includes a model acquisition module, a display interaction module, and a distance acquisition module, the model acquisition module configured to acquire a three-dimensional model of a target awaiting reconstruction. and a 3D model is obtained by the 3D reconstruction device in the fourth aspect, the display interaction module is configured to receive a plurality of measurement points set by the user on the 3D model, and the distance acquisition module is configured to: The apparatus is configured to obtain distances between a plurality of measurement points and obtain distances between positions corresponding to the plurality of measurement points on a target awaiting reconstruction.

An electronic device according to a seventh aspect of the present disclosure includes a memory and a processor coupled to each other, and the processor implements the three-dimensional reconstruction method according to the first aspect, or performs the three-dimensional reconstruction method according to the second aspect. or the measurement method based on three-dimensional reconstruction according to the third aspect, the method is configured to execute program instructions stored in the memory.

A computer-readable storage medium according to an eighth aspect of the present disclosure stores program instructions, and when the program instructions are executed by a processor, realizes the three-dimensional reconstruction method according to the first aspect, or implements the three-dimensional reconstruction method according to the second aspect. An interaction method based on three-dimensional reconstruction is realized, or a measurement method based on three-dimensional reconstruction in the third aspect is realized.

A computer program according to a ninth aspect of the present disclosure includes a computer readable code, and when the computer readable code operates on an electronic device and is executed by a processor of the electronic device, the computer program performs the three-dimensional reconstruction method according to the first aspect. or the interaction method based on the three-dimensional reconstruction in the second aspect, or the measurement method based on the three-dimensional reconstruction in the third aspect.

A computer program product according to a tenth aspect of the present disclosure, when operating on a computer, causes the computer to perform the three-dimensional reconstruction method according to the first aspect, or is based on the three-dimensional reconstruction method according to the second aspect. An interaction method is executed, or a measurement method based on three-dimensional reconstruction in the third aspect is executed.

In the above solution, the pose optimization parameters of each dataset may be determined based on the pose parameters of the dataset before it, and thus the pose optimization parameters of each dataset are used to When adjusting the camera pose parameters of the pending image to which the included image data belongs, it is useful to eliminate errors in the camera pose parameters accumulated during the scanning process, so after adjusting the predetermined 3D reconstruction method and the pending image. The reconstruction process is performed on the image data of the image to be processed using the camera pose parameters of By eliminating errors in camera pose parameters, the amount of calculation can be reduced, which helps reduce the calculation load.

2 is a flowchart of an embodiment of the three-dimensional reconstruction method of the present disclosure.

FIG. 2 is a schematic diagram of a state of an embodiment of the three-dimensional reconstruction method of the present disclosure.

2 is a flowchart of an embodiment of step S12 in FIG. 1. FIG.

2 is a flowchart of an embodiment of step S13 in FIG. 1. FIG.

2 is a flowchart of an embodiment of step S14 in FIG. 1. FIG.

6 is a flowchart of an embodiment of step S141 in FIG. 5. FIG.

6 is a flowchart of an embodiment of step S142 in FIG. 5. FIG.

6 is a flowchart of an embodiment of step S143 in FIG. 5. FIG.

2 is a flowchart of one embodiment of the 3D reconstruction-based interaction method of the present disclosure.

1 is a flowchart of one embodiment of a measurement method based on three-dimensional reconstruction of the present disclosure.

1 is a schematic framework diagram of an embodiment of a three-dimensional reconstruction device of the present disclosure; FIG.

1 is a framework schematic diagram of one embodiment of an interaction device based on three-dimensional reconstruction according to the present disclosure; FIG.

1 is a framework schematic diagram of one embodiment of a measurement device based on three-dimensional reconstruction of the present disclosure; FIG.

1 is a schematic framework diagram of an embodiment of an electronic device of the present disclosure; FIG.

1 is a framework schematic diagram of one embodiment of a computer-readable storage medium of the present disclosure; FIG.

The solutions of the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings of this specification.

In the following description, by way of explanation and not limitation, specific details, such as specific system structures, interfaces, techniques, etc., are set forth to provide a thorough understanding of this disclosure.

The terms "system" and "network" are always used interchangeably herein. As used herein, the term "at least one" is only used to describe an association relationship of associated objects, and indicates that three types of relationships can exist, e.g., at least one of A and B is , A exists alone, A and B exist simultaneously, and B exists alone. Further, in this specification, the character "/" generally indicates that the related objects before and after the character have an "or" relationship. Moreover, "plurality" in this specification means two or more than two.

Currently, 3D reconstruction is an important problem in the computer vision and augmented reality fields, and plays an important role in applications such as augmented reality on mobile platforms, games and 3D printing. When realizing real-world AR effects, such as skeletal actuation, on mobile platforms, users are usually required to quickly reconstruct the real objects in three dimensions, and therefore, three-dimensional object scanning and reconstruction techniques are highly effective on mobile platforms. is widely needed in the augmented reality field.

However, real-time 3D object reconstruction on mobile platforms has many problems. (1) Accumulated pose errors of the photographic device are eliminated. Here, there are different degrees of noise in the deep video stream and the image video stream obtained from the shooting device, and the current mainstream camera pose parameter solving method inevitably causes some errors, and the errors are It continues to accumulate as the scan progresses, thus affecting the final model effect. (2) Since there are large differences in various aspects such as color, size, and shape of objects awaiting reconstruction, robustness and applicability of the reconstruction method are highly required. , the reconstruction method is required to have an incremental reconstruction mode, with the expansion of the scanning field of view, newly taken images are continuously incorporated, and new regions of the model entering the field of view are continuously integrated with the existing model. The computational load of the entire reconstruction process increases, and the computing resources of mobile platforms are limited.

In view of this, how to enhance the effect of 3D reconstruction and reduce the computational load for 3D reconstruction becomes an urgent problem to be solved. The present disclosure provides three-dimensional reconstruction and related interactions, measurement methods and related apparatus and devices. The imaging device acquires multiple frames of images to be processed obtained by scanning the target to be reconstructed, and uses the images to be processed for each frame and the calibration parameters of the imaging device to calculate the images to be processed for each frame. The target pixel point belonging to the target to be reconstructed and its camera pose parameters are determined, and the image data of the image to be processed of each frame is sequentially divided into corresponding datasets, and the image data of the dataset and the time series are The image data and pose optimization parameters of the dataset located in the dataset are sequentially used to determine the pose optimization parameters of the dataset, and the pose optimization parameters of the dataset are used to The camera pose parameters of the image to be processed to which the data belongs are adjusted, the reconstruction process is executed on the image data of the image to be processed, and a three-dimensional model of the target to be reconstructed is obtained.

In the above solution, the pose optimization parameters of each dataset may be determined based on the pose parameters of the dataset before it, and thus the pose optimization parameters of each dataset are used to When adjusting the camera pose parameters of the pending image to which the included image data belongs, it is useful to eliminate errors in the camera pose parameters accumulated during the scanning process, so after adjusting the predetermined 3D reconstruction method and the pending image. The reconstruction process is performed on the image data of the image to be processed using the camera pose parameters of By eliminating errors in camera pose parameters, the amount of calculation can be reduced, which helps reduce the calculation load.

The execution body of the 3D reconstruction method, the interaction method based on 3D reconstruction, and the measurement method based on 3D reconstruction may be an electronic device, and the electronic device may be a smartphone, a desktop computer, a tablet computer, or a notebook. It may be a type of entity device such as a computer, smart speaker, digital assistant, augmented reality (AR)/virtual reality (VR) device, smart wearable device, etc. It may be software running on the entity device, such as an application program, a browser, etc. Here, the operating system that operates on the entity device includes, but is not limited to, an Android system, an Apple system (IOS: Input Output System), Linux, Windows, and the like.
Referring to FIG. 1, FIG. 1 is a flowchart of one embodiment of the three-dimensional reconstruction method of the present disclosure. The three-dimensional reconstruction method may include the following steps:

In step S11, the imaging device acquires a plurality of frames of images to be processed, which are obtained by scanning the target to be reconstructed.

Capturing devices can include, but are not limited to, mobile terminals such as cell phones, tablet computers, and the like. The steps in embodiments of the disclosed method may be performed by a mobile terminal or by processing equipment, such as a microcomputer connected to a capturing device with scanning and capturing capabilities, and are not limited here. Not done. In one implementation scenario, the imaging device can include a color camera that can sense visible light and a depth camera that can sense the depth of the target awaiting reconstruction, such as a structured light depth camera. When the photographing device includes a color camera and a depth camera, each frame of the image to be processed includes color data and depth data.

Targets awaiting reconstruction can include, but are not limited to, humans, animals, objects (statues, furniture, etc.). For example, when using a statue as a target for reconstruction, by scanning the statue, a 3D model of the statue can be finally obtained, and based on this, the 3D model of the statue can be Operations such as rendering, osseointegration, etc. may be performed, but are not limited here, and the reconstruction pending goal may be determined according to the actual application needs, and is not limited here.

In step S12, the target pixel point belonging to the reconstruction target of the image to be processed of each frame and its camera pose parameter are determined using the image to be processed of each frame and the calibration parameters of the photographing device.

The calibration parameters may include internal parameters of the imaging device, for example, if the imaging device includes a color camera, the calibration parameters may include internal parameters of the color camera, and the imaging device includes a depth camera. This analogy can be made in the case of a case including a color camera and a depth camera, and we will not give examples one by one here. In one implementation scene, the intrinsic parameters may include, but are not limited to, the focal length of the camera, the principal point coordinates of the camera, and in one implementation scene, the intrinsic parameters may be represented in matrix form. , for example, the internal parameters of a color camera may be expressed as:

Here, in equation (1),

represents the focal length of the color camera,

represents the principal point coordinates of the color camera. Also, the internal parameters of the depth camera

may be inferred in this way, and we will not give examples one by one here.

The calibration parameters may further include extrinsic parameters between the depth camera and the color camera of the imaging device, and are used to represent the transformation from the world coordinate system to the camera coordinate system. In embodiments of the present disclosure, the extrinsic parameters include a 3*3 rotation matrix

and 3*1 translation matrix

and may include. rotation matrix

The coordinate point in the world coordinate system using

Multiply the left and the translation matrix

By summing with, the coordinate point in the world coordinate system

The corresponding coordinate point in the camera coordinate system of

can be obtained.

In the actual scanning process, it is inevitable that objects (e.g., ground, walls, etc.) that do not belong to the target awaiting reconstruction are scanned, and therefore, in order to increase the effectiveness of the subsequent 3D reconstruction, the It is necessary to determine the target pixel point belonging to the target to be reconstructed in the image to be processed. In one implementation scene, a pre-trained image segmentation model (e.g. Unet model) is used to perform image segmentation on the image to be processed, thereby determining the target pixel point belonging to the target to be reconstructed in the image to be processed. In another implementation scene, the reconstruction target can be placed in an environment with large chromatic aberration with it, for example, if the reconstruction target is a milky white plaster statue, the reconstruction target can be It can be placed in a dark environment for scanning, thereby invalidating the marking of pixel points belonging to the environment color of the pending image, validly marking the pixel points belonging to the target color pending reconstruction, and validly marking the pixel points belonging to the target color of the pending image. The sizes of connected domains made up of pixel points are compared, and the pixel point in the largest connected domain is determined as the pixel point belonging to the target awaiting reconstruction.

In order to obtain a complete 3D model of the reconstructed target, the imaging device needs to scan the reconstructed target in different poses, and therefore the camera pose parameters used to capture the different reconstructed images. Therefore, in order to eliminate errors in the camera pose parameters and enhance the effectiveness of subsequent three-dimensional reconstruction, it is first necessary to determine the camera pose parameters of the images to be processed for each frame. In one implementation scene, the target pixel point belonging to the reconstruction target of the image to be processed in each frame

and the target pixel point belonging to the reconstruction target of the image to be processed in the frame before it.

, internal parameters of the shooting device

using relative pose parameters

, and minimize the objective function using ICP (Iterative Closest Point) algorithm, thereby determining the relative pose parameters.

can be found, where the relative pose parameter

is the pending camera pose parameter of the previous frame

Camera pose parameters of each frame's pending image for

is a relative parameter of Here, the relative pose parameter

For the objective function of , we can refer to the following formula:

Among the above formulas, in formulas (2) to (5),

is the weight,

is the pixel point

(pixel point

is the target pixel point)

The two-dimensional image coordinates are

is the depth data

the color data

pixel point after projection to

is the depth value of Therefore, in the above formula,

is the pixel point of the current frame

is the relative pose parameter

, internal parameters

can be used to represent the position coordinates in three-dimensional space of the theoretical corresponding pixel point after being transformed into the previous frame, and the relative pose parameter

The more accurate is the color data of the previous frame of the corresponding pixel point

pixel value at

and pixel point

color data of the current frame of

pixel value at

sum of squared error with

becomes smaller, and the depth data of the previous frame of the corresponding pixel point

depth value of

In the three-dimensional space of pixel points corresponding to

coordinate value

sum of squares error with

is also smaller, therefore the above objective function

By optimizing the relative pose parameters

can be obtained accurately, thereby improving the accuracy of camera pose parameters.

Camera pose parameters of the pending image of the previous frame

Camera pose parameters of each frame's pending image for

relative pose parameters of

is obtained, the relative pose parameters

The opposite of (i.e.

) is the camera pose parameter of the pending image of the frame before it.

and the camera pose parameters of the pending image of the current frame and left multiplied by

can be obtained. In one implementation scene, if the pending image is the first frame of a plurality of frames of pending images obtained by scanning by the imaging device, the camera pose parameters may be initialized as a unit matrix. Often, in embodiments of the present disclosure, the identity matrix is a rectangular matrix in which the elements on the main diagonal are all 1's and the other elements are 0's. In another implementation scene, the scanning of the image to be processed and the determination of the target pixel point and camera pose parameters can be performed at the same time, that is, after one frame of the image to be processed is obtained by scanning, The target pixel point and camera pose parameters are determined for the image to be processed obtained by 3D reconstruction can be performed in real time and online.

In step S13, the image data of each frame of the image to be processed is sequentially divided into corresponding data sets according to a predetermined division policy, where the image data includes at least a target pixel point.

In one implementation scene, when dividing, it is possible to set the maximum number of frames (for example, 8 frames, 9 frames, 10 frames, etc.) of images waiting to be processed to which the image data that can be accommodated in each dataset belongs. When the number of frames of the pending image to which the image data included in the dataset belongs reaches the maximum number of frames, a new dataset is created and the image data of the undivided pending image is added to the newly created dataset. It continues to split and cycle like this until the scan is complete. In another implementation scene, image data of images that have similar poses (for example, similar camera orientation angles, similar camera positions, etc.) and are consecutive in time series and are waiting to be processed are divided into the same data set. However, there is no specific limitation here. In yet another implementation scenario, when dividing the image data of the image to be processed in each frame, the pose difference (for example, camera orientation angle difference, It is possible to determine whether the camera position (distance) is smaller than a predetermined lower limit value, and if so, ignore the image waiting to be processed to be divided and perform the dividing operation on the image data of the image waiting to be processed in the next frame. It can also be processed. In yet another implementation scene, there may be image data belonging to the same pending image between adjacent datasets, for example, there may be two frames of image data belonging to the same pending image between adjacent datasets. data may exist, or image data belonging to three frames of the same image to be processed may exist between adjacent data sets, and is not limited here.

In one implementation scenario, the image data of the pending image of each frame may only include target pixel points (target pixel points in depth data, target pixel points in color data, etc.) that belong to the target to be reconstructed, and may include other target pixel points (target pixel points in depth data, target pixel points in color data, etc.) In the implementation scene, the image data of the pending image of each frame can further include pixel points that do not belong to the target of the reconstruction pending, for example, the image data divided into datasets is the image of the entire pending image. data, in which case the image data may further include position coordinates of the target pixel point to facilitate subsequent retrieval of the target pixel point.

Referring to FIG. 2, FIG. 2 is a schematic diagram of the state of one embodiment of the three-dimensional reconstruction method of the present disclosure. As shown in FIG. 2, the target to be reconstructed is a plaster sculpture of a human figure, and the image to be processed 21 of each frame can include color data 22 and depth data 23, and the target pixels belonging to the target to be reconstructed are The points are acquired so that the image data 24 containing at least the target pixel point is sequentially divided into corresponding data sets 25.

In step S14, pose optimization parameters for each data set are determined by sequentially using the image data of each data set and the image data and pose optimization parameters of the data set whose time series is placed before it.

In one implementation scene, the image data of each dataset and the image data of the dataset whose time series is placed before it are sequentially used to calculate the spatial transformation parameters between the two.

can be determined, which allows the spatial transformation parameter between the two to be

, and their respective pose optimization parameters

Pose optimization parameters using

You can construct the objective parameters for the pose optimization parameters by further solving the objective function

and the pose optimization parameters of the dataset in front of which the time series is placed, which allows the pose optimization parameters of the dataset in front of it to be obtained.

can be updated. Therefore, pose optimization parameters for each dataset

When solving sequentially, the pose optimization parameters of the dataset whose time series is placed before it

is taken into account, i.e. the pose optimization parameters of a dataset and the previous dataset are related to each other, and as new datasets are continuously generated, the pose optimization parameters of the previous dataset are also updated continuously. and thus cycle through to the last data set, thereby obtaining the final pose optimization parameters for each data set, thus effectively eliminating the accumulated error. In one implementation scenario, if the dataset is a first dataset, the pose optimization parameters for the first dataset may be initialized as an identity matrix. In one implementation scene, each time a new dataset is created, the pose optimization parameters of the previous dataset can be calculated, and the update of the pose optimization parameters of the related datasets can be realized. and continues until the end of the scan, yielding the final pose optimization parameters for each dataset, which can help equalize the computational effort and further reduce the computational load. If the shooting device is a mobile terminal such as a mobile phone or tablet computer, the captured images to be processed can be divided into corresponding datasets, and the pose optimization parameters of the datasets can be solved and updated. 3D reconstruction of a target waiting for reconstruction can be performed in real time and online. It should be noted that in embodiments of this disclosure and other disclosed embodiments described below, unless otherwise specified, the time series may represent the overall capture time series of images to be processed within the dataset. For example, dataset 1 includes an image to be processed in the shooting time series t=1, an image to be processed in the shooting time series t=2, and an image to be processed in the shooting time series t=3, and dataset 2 includes images to be processed in the shooting time series t=3. The overall shooting time series of the images waiting to be processed in dataset 1 includes images to be processed in series t=4, images to be processed in shooting time series t=5, and images to be processed in shooting time series t=6. The time series of Data Set 1 can be considered to precede Data Set 2 if it is located in the overall capture time series of the pending images in Data Set 2. In other cases, analogies can be made like this, and we will not give examples one by one here.

In one implementation scene, referring to FIG. 2, image data in dataset 25 is used to realize dynamic adjustment to the scanning process, improve user experience, and reduce computational load for 3D reconstruction. A set of image data can be sequentially mapped into a three-dimensional space to obtain a three-dimensional point cloud corresponding to each data set.

In one implementation scene, the camera pose parameters of the pending image to which the image data belongs and the internal parameters of the imaging device are

can be used to map the image data into a 3D space and obtain a 3D point cloud, firstly, the image data can be made 3D homogeneous, the pixel coordinates can be obtained, and the camera pose parameters

and internal parameters

can be used to left-multiply the pixel coordinates after they become homogeneous to obtain a three-dimensional point cloud in three-dimensional space. In another implementation scene, pose optimization parameters for the dataset

can be used to left-multiply the 3D point cloud and realize dynamic adjustment. That is, after the pose optimization parameters for each dataset are obtained, the camera pose parameters of the dataset can be used to adjust its corresponding three-dimensional point cloud. In yet another implementation scenario, the three-dimensional point cloud may be shown in a predetermined color (eg, green), but is not limited here.

In step S15, the pose optimization parameters of each data set are used to adjust the camera pose parameters of the image to be processed to which the image data included in the data set belongs.

Here, pose optimization parameters for each dataset

is used to determine the camera pose parameters of the pending image to which the image data contained therein belongs.

can be left multiplied, thereby realizing the adjustment of camera pose parameters. For example, the time series divided into dataset A in dataset A are image data 01 (belongs to image 01 waiting to be processed), image data 02 (belongs to image 02 waiting to be processed), image data 03 (belongs to image 03 waiting to be processed). ), the pose optimization parameters of dataset A

The camera pose parameters of image 01 to be processed using

, camera pose parameters of image 02 to be processed

, camera pose parameters of image 03 to be processed

are respectively left multiplied, thereby realizing adjustment of the camera pose parameters of the images to be processed to which the image data included in the data set A belongs. In one implementation scene, if there is image data belonging to the same pending image between adjacent datasets, only the pose optimization parameters of either of the two datasets are used to determine the camera pose of the same pending image. Parameters can be adjusted. For example, if we still take the above data set A as an example, the data set B adjacent to it includes image data 03 (belonging to the image to be processed 03) and image data 04 (belonging to the image to be processed 04), so the data set A Pose optimization parameters for

The camera pose parameters of image 01 to be processed using

, camera pose parameters of image 02 to be processed

, camera pose parameters of image 03 to be processed

When adjusting the camera pose parameters of the pending image to which the image data included in dataset B belongs, the pose optimization parameters of dataset B

The camera pose parameters of image 03 to be processed using

It is possible to perform left multiplication by the camera pose parameter of the image to be processed 04 without performing left multiplication.

In one implementation scene, referring to Figure 2, after the pose optimization parameters of each dataset are obtained, the pose optimization parameters of each dataset are used to determine the pending processing to which the image data included in the dataset belongs. The camera pose parameter 26 of the image is adjusted, and the adjusted camera pose parameter 27 is obtained. After each dataset's adjusted pose optimization parameters 27 are obtained, the dataset's adjusted camera pose parameters 27 can be used to adjust its corresponding 3D point cloud 28, thereby , the user can feel the dynamic adjustment of the 3D point cloud.

In step S16, a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the image to be processed are used to execute the reconstruction process on the image data of the image to be processed, and the three-dimensional reconstruction target of the image to be processed is Obtain a dimensional model.

The predetermined three-dimensional reconstruction method may include, but is not limited to, a TSDF (Truncated Signed Distance Function) reconstruction method and a Poisson reconstruction method. The TSDF reconstruction method is a method for calculating hidden surfaces through 3D reconstruction, and a detailed explanation thereof will be omitted here. The core idea of Poisson reconstruction is that the 3D point cloud represents the surface position of the target awaiting reconstruction, and its normal vectors represent the inner and outer directions, implicitly representing the indicator function derived from one object. A single smooth object surface estimation can be obtained by fitting to , and a detailed description thereof will be omitted. In one implementation scenario, when the imaging device captures a target to be reconstructed, the above steps reconstruct a 3D model of the target to be reconstructed in real time at the same position and angle as the currently captured image frame. They can be rendered in superimposition, thereby displaying to the user the main three-dimensional model of the target awaiting reconstruction. In another implementation scenario, a three-dimensional printer can be used to print the three-dimensional model reconstructed in the above steps to obtain a physical model corresponding to the target awaiting reconstruction.

In the above solution, the pose optimization parameters of each dataset may be determined based on the pose parameters of the dataset before it, and thus the pose optimization parameters of each dataset are used to When adjusting the camera pose parameters of the pending image to which the included image data belongs, it is useful to eliminate errors in the camera pose parameters accumulated during the scanning process, so after adjusting the predetermined 3D reconstruction method and the pending image. The reconstruction process is performed on the image data of the image to be processed using the camera pose parameters of By eliminating errors in camera pose parameters, the amount of calculation can be reduced, which helps reduce the calculation load.

Referring to FIG. 3, FIG. 3 is a flowchart of one embodiment of step S12 of FIG. Here, FIG. 3 is a flowchart of the target pixel point determination process of FIG. The determination process may include the following steps.

In step S121, the included angle between the normal vector of each pixel point included in the depth data after alignment with the color data and the gravity direction of the image to be processed is obtained.
Here, the image waiting to be processed for each frame is the color data

and depth data

and depth data

the color data

Depth data after being projected and aligned

can be obtained. In one implementation scene, we use equation (6) to calculate the depth data

Two-dimensional image coordinates of pixel points in

its depth value

3-dimensional homogeneous coordinates

Can be converted to:

Based on equation (7), the internal parameters of the depth camera in the imaging device

3D homogeneous coordinates using

After back-projecting into 3D space, the rotation matrix of depth camera and color camera is

and translation matrix

Perform a stiffness transformation to obtain the internal parameters of the color camera using

is projected onto a two-dimensional plane, and the coordinates of the target pixel point in the color data are

Get:

In the above formula, the target pixel point coordinates in color data

is a three-dimensional coordinate, and in order to convert it into a two-dimensional coordinate, its depth value, i.e. the third value, is calculated based on equation (8).

and divide its first value by its second value to obtain the coordinates of the pixel point of interest in the color data.

two-dimensional coordinates of

Get:

Further, a predetermined floating point number (for example, 0.5) can be added to each of the above division results, but the explanation will be omitted here.

In three-dimensional space, since one plane can be determined by any three points that are not located on the same line, it is possible to obtain a vector perpendicular to the plane, and therefore the normal vector of each pixel point is , by determining one plane with two pixel points adjacent to it, and further solving for a plane perpendicular to the plane. In order to improve the accuracy of the normal vector of each pixel point, obtain multiple neighboring pixel points (for example, 8 neighboring pixel points) of each pixel point, and add any two of the plurality of neighboring pixel points and each pixel point. It is possible to determine one plane in three-dimensional space using each of , solve for the vector perpendicular to the plane, and finally calculate the average value of the plurality of vectors as the normal vector of each pixel point. pixel point

For example, its depth value

Its 3D homogeneous coordinates can be obtained based on the internal parameters

is left multiplied by the three-dimensional homogeneous coordinates, and the pixel point is

is a 3D point that is back-projected into 3D space.

You can get the pixel point

The eight neighboring pixel points in the 3*3 window are arranged counterclockwise, each is back-projected into the three-dimensional space, and the corresponding three-dimensional points are obtained.

, the pixel point

3D normal vector of

may be expressed as:

In the above formula, in formula (9),

represents the cross product,

represents taking the remainder, for example,

represents the remainder when 1 is divided by 8, that is, 1, and other cases can be inferred in this way, so we will not give examples one by one here.

In one implementation scene, the included angle between the normal vector and the direction of gravity may be calculated using a cosine formula, and the description will be omitted here.

In step S122, each pixel point is projected in the direction of gravity in three-dimensional space, and the height value of each pixel point in three-dimensional space is obtained.

Still a pixel point

For example, refer to the steps above to find a 3D point in 3D space.

Get the 3D point

Obtain the projected 3D point in the gravitational direction of

The height value in 3D space of

It can be used as

The step of calculating the included angle between the normal vector of each pixel point and the gravitational direction of the image to be processed in step S121 and the step of calculating the height value of each pixel point in the three-dimensional section in step S122 are performed in order. or may be executed simultaneously, and is not limited here.

In step S123, the height value of the pixel point whose included angle satisfies a predetermined angle condition is analyzed to obtain the plane height of the target awaiting reconstruction.

In one implementation scene, the predetermined angle condition includes that the included angle between the normal vector of the pixel point and the gravitational direction of the image to be processed is less than or equal to a predetermined angle threshold (e.g., 15 degrees, 10 degrees, etc.). Therefore, based on the included angle corresponding to each pixel point obtained in step S121 above, screening is performed according to a predetermined angle condition, pixel points satisfying the condition are obtained, and then the angle obtained in step S122 is From the height value of each pixel point in 3D space, the height value of the pixel point that satisfies the above predetermined angle condition can be queried, and the height value of the pixel point that satisfies the above predetermined angle condition can be queried. You can use the values as a height set and then perform a cluster analysis on the height values in the height set to obtain the planar height of the target awaiting reconstruction, which allows the height By simply using the value of , it is possible to obtain the planar height of the target awaiting reconstruction, and the calculation load can be reduced. In one implementation scenario, when performing cluster analysis, the Random Sample Consensus (RANSAC) algorithm is used to cluster the height set and randomly set one height value, the current plane height, each time. , the number of interior points whose height difference from the plane height is within a predetermined drop range (for example, 2 cm) is calculated, and if the number of interior points or the number of repetitions satisfies the predetermined clustering condition, all Average the height values of the interior points as one candidate height and perform the next clustering on the remaining height values in the height set to ensure that the quantity in the height set is less than a given threshold. If there are multiple candidate heights, the candidate height with the smallest value and the corresponding number of interior points at a predetermined threshold can be selected as the final planar height.

In step S124, target pixel points belonging to the object to be reconstructed in the color data are screened based on the plane height.

Here, we screen pixel points whose height value is greater than the plane height, query the pixel points corresponding to the screened pixel points in the color data as candidate pixel points, and select the maximum A connected domain can be determined, and candidate pixel points in the maximum connected domain can be used as target pixel points belonging to the target awaiting reconstruction.

Unlike the above embodiment, it is possible to automatically identify the target pixel point belonging to the target to be reconstructed in the image to be processed in each frame in combination with the gravity direction, and reduce the computational load of three-dimensional reconstruction. And user intervention can be avoided, thus improving the user experience.
Referring to FIG. 4, FIG. 4 is a flowchart of one embodiment of step S13 in FIG. Here, FIG. 4 is a flowchart of an embodiment of dividing the image data of the image to be processed in each frame into corresponding data sets. It may include the following steps:

In step S131, the images to be processed of each frame are sequentially used as the current images to be processed.

Here, when the image data of a certain frame of an image to be processed is divided, it can be used as the current image to be processed.

In step S132, when dividing the image data of the current image to be processed, it is determined whether the last data set among the existing data sets satisfies a predetermined overflow condition, and if so, step S133 is executed. , otherwise, step S134 is executed.

There may be only one existing dataset, and this dataset is the last one, or there may be multiple existing datasets, and this dataset is the last one. It is the latest created of the set. For example, the existing datasets are dataset A, dataset B, dataset C, and if dataset C is created the latest, dataset C can be used as the last dataset.

In one implementation scenario, the dataset can be constructed adaptively according to the number of frames, the angle and position changes of the imaging device, making the construction of the dataset more robust, and a predetermined overflow condition can be applied to the final dataset. The number of frames of the image waiting to be processed corresponding to the included image data is greater than or equal to a predetermined frame number threshold (e.g., 8 frames, 9 frames, 10 frames, etc.), and the process to which any of the image data in the last data set belongs. The distance between the camera position of the pending image and the camera position of the current pending image is greater than a predetermined distance threshold (for example, 20 cm, 25 cm, 30 cm, etc.), and the pending image data of any of the last datasets belongs to The difference between the camera orientation angle of the image and the camera orientation angle of the current pending image may be greater than a predetermined angle threshold (eg, 25 degrees, 30 degrees, 35 degrees, etc.). Here, the camera orientation angle and camera position can be calculated using camera pose parameters of the image to be processed. Here, camera pose parameters

is the matrix

, i.e. the camera pose parameters may be expressed as the rotation matrix

and translational matrix

including camera position

may be expressed as:

In the above formula, in formula (10),

represents the transposition of the matrix. Also,

The vector in the third row of can be expressed as a camera orientation angle.

In step S133, the image data of the latest plurality of frames in the last data set are acquired and stored in the newly created dataset as a new last data set, and the image data of the current image to be processed is transferred to the new last data set. into datasets.

Still taking the existing dataset A, dataset B, and dataset C as an example, when dividing image data 10 (belonging to image 10 to be processed), the current last dataset C satisfies the predetermined overflow condition. Then, the image data of the latest plurality of frames of images waiting to be processed in the last dataset C are acquired. image data 07 (belongs to image 07 to be processed), image data 08 (belongs to image 08 to be processed), and image data 09 (belongs to image 09 to be processed); It is possible to obtain the image data of waiting image 07 to processing waiting image 09, or it is also possible to obtain the image data of processing waiting image 08 to processing waiting image 09 among them. For example, the image data of image 07 to be processed to image 09 to be processed is stored in the data set D. At this time, the image data 07 (to be processed) image data 08 (belongs to image 07 to be processed), image data 09 (belongs to image 09 to be processed) are included in chronological order, and using dataset D as the new last dataset, the image Data 10 (belonging to image 10 to be processed) is divided into data set D.

In one implementation scenario, when dividing the image data of the current image to be processed, there is a possibility that the last data set does not satisfy a predetermined overflow condition, and in this case, step S134 below can be executed.

In step S134, the image data of the current image to be processed is divided into the final data set.

Still taking the existing dataset A, dataset B, and dataset C as an example, when dividing image data 10 (belonging to image 10 to be processed), it is assumed that the current last dataset C satisfies the predetermined overflow condition. If not, the image data 10 (belonging to the image to be processed 10) is divided into the final data set C.

Unlike the above embodiment, when dividing the image data of the current pending image, when the last dataset of the existing datasets satisfies a predetermined overflow condition, the latest multiple frames in the last dataset Get the image data of the pending image and store it in the newly created dataset as the new last dataset, so there are multiple frames of image data of the same pending image between adjacent datasets, and the adjacent This helps to improve the alignment effect between data sets, and further helps to improve the effect of three-dimensional reconstruction.
Referring to FIG. 5, FIG. 5 is a flowchart of one embodiment of step S14 in FIG. Here, FIG. 5 is a flowchart of one embodiment for determining pose optimization parameters for a dataset. It may include the following steps:

In step S141, each data set is sequentially used as a current data set, and at least one data set whose time series is placed before the current data set is selected as a candidate data set.

Still taking the above existing dataset A, dataset B, dataset C as an example, when determining the pose optimization parameters of dataset B, dataset B can be used as the current dataset, and dataset When determining pose optimization parameters for C, dataset C can be used as the current dataset. Furthermore, when creating a new dataset, pose optimization parameters of the dataset before the newly created dataset can be determined. For example, as in the above embodiment, image data 10 (image 10 ), when the current last dataset C satisfies a predetermined overflow condition, a new dataset D is newly created, using dataset C as the current dataset, and its pose Optimization parameters can be determined.
In one implementation scene, in order to improve the accuracy of pose optimization parameters and thereby enhance the 3D reconstruction effect, image data with similar image data is extracted from a dataset placed before the current dataset. Referring now to FIG. 6, FIG. 6 is a flowchart of one embodiment of step S141 of FIG. 5, which may include the following steps:

In step S61, a Bag of Words model is constructed using predetermined image features of the image data of the current dataset and the dataset before which the time series is placed.

The predetermined image features may include ORB (Oriented FAST and Rotated BRIEF) image features, allowing feature vectors to be quickly created for key points in the image data, and the feature vectors Fast and Brief are a feature detection algorithm and a vector creation algorithm, respectively, which may be used to identify a waiting target, and detailed description thereof will be omitted here.

The Bag of Words model is a simplified representation model under natural language processing and information retrieval, and each predetermined image feature in the Bag of Words model is independent, and here Detailed explanation will be omitted. In one implementation scenario, when starting to create a new dataset, use the previous dataset as the current dataset, extract certain image features of the image data of the current dataset, and can be added to the Bag of Words model, and when cycled in this way, the Bag of Words model may be expanded step by step. In one implementation scene, there is superimposed image data between the current dataset and the previous dataset, so when extracting predetermined image features of the image data in the current dataset, Feature extraction is not performed on the image data superimposed with the dataset.

In step S62, image data in a predetermined time series of the current data set of images to be processed to which it belongs is selected as image data to be matched.

In one implementation scene, the predetermined time series can include a beginning, a middle, and an end, still taking the dataset C in the above example as an example, the dataset C includes image data 05 (belonging to the pending image 05). ), image data 06 (belongs to image 06 to be processed), image data 07 (belongs to image 07 to be processed), image data 08 (belongs to image 08 to be processed), and image data 09 (belongs to image 09 to be processed). It is possible to select the image data 05 of the image 05 to be processed at the beginning, the image data 07 of the image 07 to be processed in the middle, and the image data 09 of the image 09 to be processed at the end as the image data to be matched. For other implementation scenes, analogy can be made in this way, and examples will not be given here one by one. In addition, the predetermined time series may be set at the beginning, 1/4 time series, 1/2 time series, 3/4 time series, or last depending on the actual situation, and is not limited here. .

In step S63, a predetermined image feature whose similarity score with the predetermined image feature of the image data awaiting matching is greater than a predetermined similarity threshold is queried from a predetermined range of the Bag of Words model.

The predetermined range may include predetermined image features of the image data to which the data set it belongs is not adjacent to the current data set and is not included in the current data set. Still taking Dataset A, Dataset B, and Dataset C in the above example as an example, if the current dataset is Dataset C, the predetermined range includes the predetermined image features and data belonging to Dataset A. and predetermined image features belonging to set B. In one implementation scenario, the predetermined similarity threshold may be one predetermined score value, such as, but not limited to, 0.018, 0.019, 0.020, etc. In another implementation scene, the maximum score value of the similarity scores between each image data and the image data waiting for matching in the current dataset and adjacent datasets

Get the maximum score value

A predetermined multiple of (eg, 1.5 times, 2 times, 2.5 times) may be used as a predetermined similarity threshold. In another implementation scene, the maximum score value

A predetermined multiple of the above predetermined score values can be used as a predetermined similarity threshold, i.e., from a predetermined range of the Bag of Words model to a predetermined image feature of the image data to be matched. Similarity score

is the maximum score value

A predetermined image feature that is greater than a predetermined multiple of any of the above predetermined score values can be queried, but is not limited to this.

In step S64, the data set in which the image data to which the queried predetermined image feature belongs is located, and the data set adjacent to the current data set are used as candidate data sets.

For example, assuming that the current dataset is Dataset H, query Dataset C and Dataset D using the image data waiting for matching at the beginning, and using the image data waiting for matching in the middle, When querying Dataset D and Dataset E, and then querying Dataset E and Dataset F using the last image data waiting for matching, Datasets C to F and Dataset G are used as the current Dataset H. can be used as a candidate dataset. In one implementation scene, a predetermined number of data sets (e.g., two, three, etc.) with the highest similarity scores are selected from the data set in which the image data to which the queried predetermined image feature belongs is located, and the current Data sets adjacent to the data set can be selected as candidate data sets. For example, if the current dataset is still dataset H, then similarity scores from datasets C to F are calculated.

The three largest datasets of G and the current dataset and adjacent datasets G can be selected as candidate datasets.

In step S142, spatial transformation parameters between the current dataset and the candidate dataset are determined using the image data of the current dataset and the image data of the candidate dataset.

In one implementation scene, the current By combining image features and positions in three-dimensional space of the image data of the dataset and candidate dataset, spatial transformation parameters between the two can be determined. Referring to FIG. 7, FIG. 1 is a flowchart of one embodiment, which may include the following steps:

In step S71, one group of image data awaiting matching that satisfies a predetermined matching condition is searched within the candidate data set and the current data set.

The predetermined matching condition may include that the difference between the camera orientation angles of the pending images to which the image data to be matched belongs is the smallest, where, for each candidate dataset, a predetermined matching condition is determined from it and the current dataset. It is possible to search for one group of matching image data that satisfies the matching condition of

, and the image data waiting for matching belonging to the candidate dataset is

can be written down.

In step S72, matching pixel point pairs between the matching image data of each group are obtained based on predetermined image features extracted from the matching image data of each group.

By combining the RANSAC algorithm,

performing matching pair screening on predetermined image features (e.g., ORB image features) of

For ease of explanation, matching pixel points between can be obtained,

You may write each. Regarding the RANSAC algorithm, the related steps in the above embodiments can be referred to, and the explanation is omitted here.

In step S73, among the matching pixel point pairs, pixel points belonging to the current data set are mapped in a three-dimensional space, a first three-dimensional matching point is obtained, and the pixel points belonging to the current data set among the matching pixel point pairs are mapped to the candidate data set. The belonging pixel points are mapped in a three-dimensional space to obtain a second three-dimensional matching point.

to the 3D space and obtain the first 3D matching point for ease of explanation.

written in

to the 3D space and obtain the second 3D matching point for ease of explanation.

Write down. here,

are converted into three-dimensional homogeneous coordinates, and the internal parameters are

the opposite of

using,

The first three-dimensional matching point is obtained by multiplying the three-dimensional homogeneous coordinates of

and second 3D matching point

can be obtained.

In step S74, the first three-dimensional matching point and the second three-dimensional matching point are aligned to obtain a spatial transformation parameter.

Here, it is possible to align the first three-dimensional matching point and the second three-dimensional matching point in the three-dimensional space, increase the degree of matching between the two as much as possible, and thereby obtain the spatial transformation parameter between the two. can. In one implementation scene, a first pose transformation parameter between a first three-dimensional matching point and a second three-dimensional matching point can be obtained, and a first pose transformation parameter between the first three-dimensional matching point and the second three-dimensional matching point can be obtained. The points are used to construct an objective function regarding the first pose transformation parameter, and the objective function is solved by a method such as SVD (Singular Value Decomposition) or nonlinear optimization to determine the first pose transformation parameter.

can be obtained.

In formula (11),

represent the i-th pair of three-dimensional matching points in the three-dimensional space, respectively.

By solving the above objective function, the first pose transformation parameter

is obtained, the first pose transformation parameter

and a predetermined pose transformation parameter (e.g., identity matrix), perform pose optimization on the first three-dimensional matching point to obtain the first optimized matching point and the second optimized matching point. and can be obtained respectively, where the first pose transformation parameter

and the first three-dimensional matching point using the predetermined pose transformation parameters

, respectively, thereby obtaining the first optimized matching point and the second optimized matching point, respectively; for ease of explanation,

can be written respectively. Furthermore, the second three-dimensional matching point

and the first optimization matching point

and the second optimization matching point

The pose transformation parameter used by the optimized matching point with a high matching degree can be selected as the second pose transformation parameter, and for ease of explanation,

can be written down. Here, the second three-dimensional matching point

and the first optimization matching point

When calculating the degree of matching with each second three-dimensional matching point

The first optimization matching point within a predetermined range (e.g., 5 cm range) of

can be searched, and if it can be searched, the second three-dimensional matching point

All second 3D matching points can be marked as valid, otherwise they can be marked as invalid.

After the search is completed, the second 3D matching point

the second 3D matching point marked as valid for the total number of

, that is, the second three-dimensional matching point

and the first optimization matching point

Calculate the degree of matching with the second three-dimensional matching point

and the second optimization matching point

The degree of coincidence may be inferred in this way, and the explanation will be omitted here.

Second pose transformation parameter

After determining the second pose transformation parameter

is the initial value, and the first three-dimensional matching point is determined by a predetermined alignment method (for example, point-to-normal ICP method)

and second 3D matching point

For ease of explanation, the spatial transformation parameters of the current dataset and candidate dataset can be obtained.

Write down. By repeating the above steps, we can determine the spatial transformation parameters between the current dataset and each candidate dataset.

can be obtained.

In step S143, the pose optimization parameters of the current dataset are obtained using at least the pose optimization parameters of the candidate dataset and the spatial transformation parameters between the current dataset and the candidate dataset; Update the pose optimization parameters of.

In one implementation scene, in order to improve the accuracy of the pose optimization parameters, the above spatial transformation parameters are calculated before solving the pose optimization parameters of the current dataset.

You can also screen the spatial transformation parameters between the current dataset and each candidate dataset to solve the pose optimization parameters for the current dataset.

From the above, spatial transformation parameters that satisfy predetermined parameter screening conditions can be selected. The predetermined parameter screening conditions are the spatial transformation parameters

the candidate dataset associated with is adjacent to the current dataset, or the spatial transformation parameters
The first 3D matching point using

The optimized matching point obtained by optimizing the pose and the second 3D matching point

This may include that the degree of agreement with the above is greater than a predetermined degree of agreement threshold (for example, 60%, 65%, 70%, etc.). Here, the spatial transformation parameter

The first 3D matching point using

can be left multiplied by , thereby realizing pose optimization to it.

Here, the pose optimization parameters of the candidate dataset and the spatial transformation parameters between the current dataset and the candidate dataset are used to construct an objective function regarding the pose optimization parameters of the current dataset, and the objective function is By solving, the pose optimization parameters of the current dataset can be obtained and the pose optimization parameters of at least the candidate dataset can be updated. Also, by cycling in this way, each dataset before the newly created dataset is used as the current dataset, and the target waiting for reconstruction is scanned to create a dataset, and the pose optimization parameters are determined. Furthermore, it is useful for equalizing the amount of calculation and reducing the calculation load, and it is possible to three-dimensionally reconstruct a target that is waiting for reconstruction in real time and online. In one implementation scene, refer to FIG. 8, which is a flowchart of one embodiment of step S143 in FIG. It may include the following steps:

In step S81, two data sets are used as a data set pair, corresponding to each spatial transformation parameter respectively associated with the current data set and the candidate data set before which the time series is placed.

As an example, assuming that the current dataset is still dataset H, datasets C to F and dataset G are used as candidate datasets for current dataset H, and the spatial transformation parameters are

The candidate data set C and the current data set H corresponding to are used as one data set pair, and the spatial transformation parameters are

The candidate dataset D and the current dataset H corresponding to are used as one dataset pair, and the spatial transformation parameters are

The candidate dataset E and the current dataset H corresponding to are used as one dataset pair, and the spatial transformation parameters are

The candidate dataset F and the current dataset H corresponding to are used as one dataset pair, and the spatial transformation parameters are

The candidate data set G corresponding to , and the current data set H are used as one data set pair. In addition, each data set before the current data set H (that is, data sets A to G) also has a corresponding spatial transformation parameter. For example, for data set B, the spatial transformation parameter between data set A and data set A is may exist, and dataset B and dataset A may be used as a dataset pair, and for dataset C, there may be spatial transformation parameters between dataset A and data B, thus , dataset C and dataset A can be used as a dataset pair, respectively, and dataset C and dataset B can be used as a dataset pair, and by analogy, we will not give examples one by one here. , for the solution of the spatial transformation parameters, the related steps in the above embodiments can be specifically referred to.

In step S82, the spatial transformation parameters of each dataset pair and the respective pose optimization parameters are used to construct an objective function for the pose optimization parameters.
Here, the objective function may be expressed as:

Here, in equation (12),

represents the number of the data set included in each data set pair (for example, expressed by alphabets such as C, D, E, or Arabic numerals such as 1, 2, 3),

represents the spatial transformation parameters between each dataset pair,

represent the pose optimization parameters of each dataset included in each dataset pair, respectively, and

represents an optimization formula, which may be expressed as:

In formula (3),

teeth,

Each represents the opposite of . Therefore, each time the spatial transformation parameters of a dataset are determined, a new optimization relationship can be introduced into the objective function, which re-optimizes the pose optimization parameters of the previous dataset and It continues until the determination of the pose optimization parameters of the set is completed, thus helping to eliminate the accumulated pose error in the scanning process, improve the accuracy of the pose optimization parameters, and enhance the effectiveness of 3D reconstruction. can. In one implementation scene, if the current dataset is the first dataset, its pose optimization parameters may be initialized as a unit matrix, and can refer to the related steps in the example above. , the explanation is omitted here.

In step S83, the objective function is solved by a predetermined solution method, and the pose optimization parameters of the datasets included in the current dataset and the datasets corresponding to each of the candidate datasets placed before the time series are determined. get.

As shown in the above equation, the pose optimization parameters of the datasets included in each dataset pair can be obtained by minimizing the objective function. Still, assuming that the current dataset is dataset H, by solving the above objective function, the pose optimization parameters of the current dataset H can be obtained, and the pose optimization parameters of datasets C to G can be further calculated. The pose optimization parameters after being optimized and the pose optimization parameters after being further optimized for the data set before the current data set H can be obtained. If we introduce a new dataset I and find the spatial transformation parameters associated with it, we can obtain the pose optimization parameters of dataset I by constructing an objective function to further optimize the previous dataset. The pose optimization parameters can be obtained after the pose optimization parameters have been determined, and such cycling can further help eliminate pose cumulative errors.

Different from the above embodiment, by sequentially using each dataset as the current dataset and selecting at least one dataset placed before the current dataset as a candidate dataset, and the image data of the candidate dataset to determine spatial transformation parameters between the current dataset and the candidate dataset, and at least pose optimization parameters of the candidate dataset and the image data of the current dataset and the candidate dataset. of the camera pose parameters accumulated in the scanning process by using the spatial transformation parameters with the dataset to obtain the pose optimization parameters of the current dataset and update the pose optimization parameters of at least the candidate dataset. It can help eliminate errors and reduce the amount of data for calculating pose optimization parameters, thereby helping to reduce the computational load.
Referring to FIG. 9, FIG. 9 is a flowchart of one embodiment of the three-dimensional reconstruction-based interaction method of the present disclosure. The three-dimensional reconstruction method may include the following steps:

In step S91, a three-dimensional model of the target awaiting reconstruction is obtained.

The three-dimensional model may be obtained by the steps in any of the three-dimensional reconstruction method embodiments described above, reference may be made to the three-dimensional reconstruction method embodiments described above, and the description is omitted here. .

In step S92, a three-dimensional map of the scene where the photographing device is placed is constructed using a predetermined visual inertial navigation method, and current pose information of the photographing device on the three-dimensional map is obtained.

The predetermined visual-inertial navigation method may include SLAM (Simultaneous Localization and Mapping), which allows a three-dimensional map on which a capturing device (e.g., mobile phone, tablet computer, etc.) is located. It is possible to construct and obtain the current pose information of the photographing device in the 3D map.

In one implementation scenario, the 3D model can also be skeletally connected to realize dynamic interaction with the 3D model. For example, if the 3D model is a quadrupedal animal such as a cow or sheep, after the 3D model is connected to the skeleton, the joints of the skeleton will be able to move according to the default rules for quadrupedal animals. Can move.

In step S93, the three-dimensional model is displayed on the scene image currently being photographed by the photographing device based on the pose information.

Here, the pose information may include the position and orientation of the photographing device. For example, if the pose information of the capture device indicates that it is facing the ground, the scene image currently being captured by the capture device may display the top of the three-dimensional model, or the pose information of the capture device represents that the orientation presents an acute angle with respect to the ground, the side surface of the three-dimensional model may be displayed in the scene image currently being captured by the imaging device. In one implementation scene, after the 3D model is skeleton-combined, driving commands input by the user may also be received, so that the 3D model can move according to the driving commands input by the user. For example, if the three-dimensional model is a sheep, the user can drive it to lower its head or walk, but is not limited here. If the three-dimensional model is a human or other object, an analogy can be made in this way, and we will not give examples one by one here.

In the above solution, the 3D model of the target awaiting reconstruction is displayed on the currently captured scene image based on the pose information of the imaging device in the 3D map of the scene, so that the virtual object and the real scene are It is possible to realize the fusion of geometrical consistency of The geometric consistency of the fusion effect can be enhanced, which helps improve the user experience.
Referring to FIG. 10, FIG. 10 is a flowchart of one embodiment of the three-dimensional reconstruction-based measurement method of the present disclosure. The three-dimensional reconstruction method may include the following steps:

In step S1010, a three-dimensional model of the target awaiting reconstruction is obtained.

The three-dimensional model may be obtained by the steps in any of the three-dimensional reconstruction method embodiments described above, reference may be made to the three-dimensional reconstruction method embodiments described above, and the description is omitted here. .

In step S1020, a plurality of measurement points set by the user in the three-dimensional model are received.

The user can set multiple measurement points on the three-dimensional model by mouse clicks, keyboard input, display touch, etc. The number of measurement points may be two, three, four, etc., and is not limited here. Referring to FIG. 2, if the target awaiting reconstruction is a plaster human figure, the user can set measurement points at the centers of both eyes of the three-dimensional model 29, or It is also possible to set measurement points for the nasal gills and the philtrum, respectively, or it is also possible to set measurement points for the centers of both eyes and the philtrum of the three-dimensional model 29, but we will not give one example here. .

In step S1030, distances between a plurality of measurement points are obtained, and distances between positions corresponding to the plurality of measurement points on the target awaiting reconstruction are obtained.

Referring to FIG. 2, if the target still waiting for reconstruction is a plaster human figure, by obtaining the distance between the centers of both eyes of the three-dimensional model 29, the distance between the centers of both eyes corresponding to the plaster human figure is calculated. Alternatively, by obtaining the distance between the nasal mound and the philtrum of the three-dimensional model 29, the distance between the nasal mound and the philtrum corresponding to the plaster figure can be obtained. Alternatively, by obtaining the distance between the centers of both eyes and the philtrum of the three-dimensional model 29, it is possible to obtain the distance between both eyes and the philtrum corresponding to the plaster human image, thereby It helps to increase the convenience of measuring objects in real scenes.

In the above solution, by receiving multiple measurement points set by the user in a 3D model, the distances between the multiple measurement points are obtained, and the positions corresponding to the multiple measurement points on the target awaiting reconstruction are obtained. 3D reconstruction, since the 3D model can be obtained by the 3D reconstruction method in the first aspect above, and the 3D reconstruction method in the first aspect above It is possible to enhance the effect of the measurement and further improve the accuracy of measurement.

Referring to FIG. 11, FIG. 11 is a flowchart of one embodiment of a three-dimensional reconstruction apparatus 1100 of the present disclosure. The three-dimensional reconstruction device 1100 includes an image acquisition section 1110, a first determination section 1120, a data division section 1130, a second determination section 1140, a parameter adjustment section 1150, and a model reconstruction section 1160. , the imaging device is configured to acquire a plurality of frames of images to be processed obtained by scanning a target to be reconstructed, and the first determination unit 1120 is configured to obtain images to be processed of each frame and the calibration of the imaging device. The data segmentation unit 1130 is configured to determine the target pixel point belonging to the reconstruction target of the image to be processed of each frame and its camera pose parameters using the motion parameters, and the data segmentation unit 1130 is configured to use The second determination unit 1140 is configured to sequentially divide the image data of the image to be processed in each frame into corresponding data sets, and the image data includes at least a target pixel point, and the second determination unit 1140 divides the image data of each data set and the time The series is configured to sequentially use the image data and pose optimization parameters of the datasets placed before it to determine pose optimization parameters for each dataset, and the parameter adjustment unit 1150 The model reconstruction unit 1160 is configured to adjust the camera pose parameters of the image to be processed to which the image data included in the dataset belongs using the pose optimization parameters of The apparatus is configured to perform reconstruction processing on image data of the image to be processed using adjusted camera pose parameters of the image to be processed, thereby obtaining a three-dimensional model of the target to be reconstructed.

In some embodiments, the second determining unit 1140 sequentially uses each data set as the current data set and selects at least one data set located before the current data set as a candidate data set. The second determining unit 1140 includes a dataset selection sub-unit configured to select between the current dataset and the candidate dataset using the image data of the current dataset and the image data of the candidate dataset. The second determining unit 1140 further includes a spatial transformation parameter sub-unit configured to determine spatial transformation parameters of the current dataset and the candidate dataset, and the second determining unit 1140 is configured to determine spatial transformation parameters of the current dataset and the candidate dataset. The method further includes a pose optimization parameter sub-portion configured to obtain pose optimization parameters of a current dataset and update pose optimization parameters of at least the candidate dataset using spatial transformation parameters.

In some embodiments, the pose optimization parameters sub-section data sets two data sets corresponding to each spatial transformation parameter respectively associated with the current data set and the data set before which the time series is placed. a dataset pair section configured for use as a set pair, and a pose optimization parameter subsection that uses the spatial transformation parameters of each dataset pair and the respective pose optimization parameters, further comprising an objective function construction unit configured to construct an objective function with respect to The method further includes an objective function solver configured to obtain pose optimization parameters for datasets included in the datasets corresponding to each of the datasets.

In some embodiments, the spatial transformation parameter subunit includes an image data search unit configured to search within the candidate dataset and the current dataset for a group of matching image data that satisfies a predetermined matching condition. , the spatial transformation parameter sub-unit is configured to obtain matching pixel point pairs between the matching image data of each group based on predetermined image features extracted from the matching image data of each group. The spatial transformation parameter sub-unit maps pixel points belonging to the current data set among the matching pixel point pairs to a three-dimensional space and obtains a first three-dimensional matching point. further comprising a three-dimensional space mapping unit configured to map pixel points belonging to the candidate data set among the matching pixel point pairs to a three-dimensional space to obtain a second three-dimensional matching point, and performing spatial transformation. The parameter sub-unit further includes a three-dimensional matching point alignment unit configured to align the first three-dimensional matching point and the second three-dimensional matching point and obtain a spatial transformation parameter.

In some embodiments, the three-dimensional matching point registration unit includes a first pose transformation parameter configured to obtain a first pose transformation parameter between the first three-dimensional matching point and the second three-dimensional matching point. The three-dimensional matching point alignment unit includes a pose transformation parameter sub-unit, and the three-dimensional matching point alignment unit performs pose optimization on the first three-dimensional matching point using the first pose transformation parameter and the predetermined pose transformation parameter. further comprising a three-dimensional matching point optimization sub-unit configured to obtain a first optimized matching point and a second optimized matching point, respectively; The degrees of correspondence between the three-dimensional matching points and the first and second optimization matching points are calculated, and the pose transformation parameters used by the optimization matching points with a high degree of agreement are calculated as The three-dimensional matching point alignment unit further includes a second pose transformation parameter sub-unit configured to select the pose transformation parameter as a pose transformation parameter, and the three-dimensional matching point alignment unit sets the second pose transformation parameter as an initial value and selects the second pose transformation parameter as an initial value according to a predetermined alignment method. The method further includes a spatial transformation parameter sub-portion configured to align the first three-dimensional matching point and the second three-dimensional matching point and obtain spatial transformation parameters of the current data set and the candidate data set.

In some embodiments, the spatial transformation parameters sub-portion is configured to select, from the spatial transformation parameters of the current dataset and each candidate dataset, spatial transformation parameters that satisfy a predetermined parameter screening condition. The screening unit further includes a predetermined parameter screening condition, wherein the candidate dataset associated with the spatial transformation parameter is adjacent to the current dataset, and the spatial transformation parameter is used to pose the first three-dimensional matching point to the optimal pose. The matching degree between the optimized matching point obtained by optimizing the matching point and the second three-dimensional matching point is greater than a predetermined matching degree threshold.

In some embodiments, the dataset selection sub-unit builds the Bag of Words model using predetermined image features of the image data of the current dataset and the dataset before which the time series is placed. further comprising a Bag of Words model construction unit configured to: select the image data located in a predetermined time series of the current dataset of the pending images to which it belongs as the image data to be matched; The dataset selection sub-unit is configured to select, from a predetermined range of the Bag of Words model, a similarity score of the image data to be matched with a predetermined image feature at a predetermined similarity threshold. further comprising an image feature query section configured to query for a larger predetermined image feature, and a dataset selection sub-section configured to select a dataset to which the image data to which the queried predetermined image feature belongs and the current data are located; configured to use the set and adjacent datasets as candidate datasets, where the predetermined range is configured such that the dataset to which it belongs is not adjacent to the current dataset and is not included in the current dataset. The image processing apparatus further includes a candidate dataset section including predetermined image features of the image data.

In some embodiments, the dataset selection sub-unit is configured to obtain a maximum score value of similarity scores between each image data and the image data to be matched in the current dataset and adjacent datasets. further comprising a maximum similarity score value obtaining unit configured to obtain a maximum similarity score value, and the dataset selection sub unit configured to use either a predetermined multiple of the maximum score value and the predetermined score value as a predetermined similarity threshold. The method further includes a similarity threshold determination unit.

In some embodiments, the data partitioning unit 1130 includes a current pending image determination subunit configured to sequentially use the pending image of each frame as the current pending image, and the data partitioning unit 1130 includes: When dividing the image data of the current pending image, when the last dataset of the existing dataset satisfies the predetermined overflow condition, the image data of the latest multiple frames of the pending image in the last dataset is obtained. and further includes a data processing sub-unit configured to store the newly created data set as a new final data set and to divide the image data of the current pending image into the new final data set.

In some embodiments, the predetermined overflow condition is that the number of frames of images to be processed that correspond to image data included in the last dataset is greater than or equal to a predetermined frame number threshold; The distance between the camera position of the image to be processed to which the image data belongs and the camera position of the current image to be processed is greater than a predetermined distance threshold, and the camera orientation of the image to be processed to which any of the image data of the last dataset belongs The difference between the angle and the camera orientation angle of the current image to be processed is greater than a predetermined angle threshold, where the camera position and camera orientation angle use the camera pose parameters of the image to be processed. It was calculated as follows.

In some embodiments, the image to be processed for each frame includes color data and depth data, and the first determining unit 1120 determines the modulus of each pixel point included in the depth data after alignment with the color data. The first determination unit 1120 includes an included angle acquisition sub-unit configured to acquire an included angle between the line vector and the gravity direction of the image to be processed, and the first determination unit 1120 projects each pixel point in the three-dimensional space in the gravity direction, The first determination unit 1120 further includes a height acquisition sub-unit configured to acquire the height value of the pixel point in three-dimensional space, and the first determining unit 1120 determines the height of the pixel point whose included angle satisfies a predetermined angle condition. The first determining unit 1120 further includes a height analysis sub-unit configured to analyze the value and obtain the planar height of the reconstruction target, and the first determining unit 1120 determines the reconstruction target of the color data according to the planar height. It further includes a pixel screening sub-unit configured to screen target pixel points belonging to the object.

In some embodiments, the height analysis sub-unit includes a height set acquisition unit configured to use, as a height set, height values of pixel points whose included angles satisfy a predetermined angular condition; The height analysis sub-unit includes a height cluster analysis unit configured to perform a cluster analysis on the height values in the height set to obtain a planar height of the awaiting reconstruction target.

In some embodiments, the 3D reconstruction device 1100 is configured to sequentially map the image data of each dataset into a 3D space to obtain a 3D point cloud corresponding to each dataset. Further comprising a dimensional mapping unit, the three-dimensional reconstruction device 1100 further comprises a point cloud adjustment unit configured to use the pose parameters of each dataset to adjust its corresponding three-dimensional point cloud.

Referring to FIG. 12, FIG. 12 is a framework schematic diagram of one embodiment of an interaction device based on three-dimensional reconstruction according to the present disclosure. The interaction device 1200 based on 3D reconstruction includes a model acquisition unit 1210, a mapping and positioning unit 1220, and a display interaction module 1230, and the model acquisition unit 1210 is configured to acquire a 3D model of a target awaiting reconstruction. , a 3D model is obtained by the 3D reconstruction device in any of the embodiments of the 3D reconstruction device described above, and the mapping and positioning unit 1220 has the imaging device arranged according to a predetermined visual inertial navigation method. The display interaction module 1230 is configured to construct a three-dimensional map of the scene and obtain current pose information of the photographing device in the three-dimensional map, and the display interaction module 1230 displays the scene image currently being photographed by the photographing device based on the pose information. is configured to display a three-dimensional model.

Referring to FIG. 13, FIG. 13 is a framework schematic diagram of one embodiment of a measurement device 1300 based on three-dimensional reconstruction. The measurement device 1300 based on three-dimensional reconstruction includes a model acquisition unit 1310, a display interaction unit 1320, and a distance acquisition unit 1330, and the model acquisition unit 1310 is configured to acquire a three-dimensional model of a target awaiting reconstruction; A 3D model is obtained by the 3D reconstruction device in any of the embodiments of the 3D reconstruction device described above, and the display interaction unit 1320 is configured to receive a plurality of measurement points set by the user in the 3D model. The distance acquisition acquisition unit 1330 is configured to acquire distances between a plurality of measurement points and to acquire distances between positions corresponding to the plurality of measurement points on the target awaiting reconstruction.

Referring to FIG. 14, FIG. 14 is a framework schematic diagram of one embodiment of an electronic device 1400 according to the present disclosure. Electronic device 1400 includes a memory 1410 and a processor 1420 coupled to each other, and processor 1420 implements the steps in any of the three-dimensional reconstruction method embodiments described above or performs any of the three-dimensional reconstruction methods described above. or executing program instructions stored in memory 1410 to implement the steps in the embodiments of the interaction method based on 3D reconstruction or to implement the steps in the embodiment of the measurement method based on any of the three-dimensional reconstructions described above. It is composed of In one implementation scene, the electronic equipment may include a mobile terminal such as a mobile phone, a tablet computer, or the electronic equipment may be a data processing device (such as a microcomputer) connected to the imaging device. , is not limited here.

Processor 1420 may also be called a CPU (Central Processing Unit). Processor 1420 may be an integrated circuit chip with signal processing functionality. The processor 1420 is a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). Mable Gate Array) or other programmable logic The device may be a discrete gate or transistor logic device, a discrete hardware component. A general purpose processor may be a microprocessor, or it may be any conventional processor, or the like. Processor 1420 may also be jointly implemented by an integrated circuit chip.

The above solution can enhance the effect of three-dimensional reconstruction and reduce the computational load for three-dimensional reconstruction.

Referring to FIG. 15, FIG. 15 is a framework schematic diagram of one embodiment of a computer readable storage medium 1500 of the present disclosure. Computer-readable storage medium 1500 stores program instructions 1501 that can be executed by a processor to implement steps in an embodiment of any three-dimensional reconstruction method described above, or to perform steps in any three-dimensional reconstruction method embodiment described above. It is configured to implement the steps in an embodiment of an interaction method based on dimensional reconstruction, or to implement the steps in an embodiment of a measurement method based on any three-dimensional reconstruction described above.

The above solution can enhance the effect of three-dimensional reconstruction and reduce the computational load for three-dimensional reconstruction.

It should be understood that in some embodiments provided in this disclosure, the disclosed methods and apparatus may be implemented in other ways. For example, the above device embodiments are only exemplary, for example, the division of modules or units is only logical and functional division, and there may be other division schemes when actually implemented, e.g. The units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Also, the mutual or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interface, device or unit, electrical, mechanical or otherwise. It may be in the form of

Units described as separate members may or may not be physically separate, and members designated as units may be physical units or physically separate. It may not be a unit, ie it may be located at one location, or it may be distributed over network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Additionally, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may be physically present alone, or two or more units may be integrated into one processing unit. , may be integrated into one unit. The above integrated unit may be realized in the form of hardware or in the form of a software functional unit.

The integrated unit may be stored on a single computer-readable storage medium when it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present disclosure may be realized essentially or in a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product, and the technical solution of the present disclosure may be embodied in the form of a software product. The software product is stored on a single storage medium and configured to perform all or some of the method steps of each embodiment of the present disclosure on a computing device (which may be a personal computer, a server, or a network device, etc.) or a processor. Contains several instructions for executing the section. The storage medium may include various media capable of storing program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. include.

In an embodiment of the present disclosure, the imaging device acquires a plurality of frames of images to be processed obtained by scanning a target to be reconstructed, and uses the images to be processed for each frame and the calibration parameters of the imaging device. , determine the target pixel point belonging to the reconstruction target of the image to be processed in each frame and its camera pose parameters, sequentially divide the image data of the image to be processed in each frame into corresponding datasets, and divide the image data of the dataset into , and using the image data and pose optimization parameters of the dataset before which the time series is placed, determine the pose optimization parameters of the dataset, and using the pose optimization parameters of the dataset, The camera pose parameters of the image to be processed to which the image data included in the dataset belongs are adjusted, and the reconstruction process is executed on the image data of the image to be processed to obtain a three-dimensional model of the target to be reconstructed. The above solution can enhance the effect of three-dimensional reconstruction and reduce the computational load for three-dimensional reconstruction.

In the above solution, the pose optimization parameters of each dataset may be determined based on the pose parameters of the dataset before it, and thus the pose optimization parameters of each dataset are used to When adjusting the camera pose parameters of the pending image to which the included image data belongs, it is useful to eliminate errors in the camera pose parameters accumulated during the scanning process, so after adjusting the predetermined 3D reconstruction method and the pending image. The reconstruction process is performed on the image data of the image to be processed using the camera pose parameters of By eliminating errors in camera pose parameters, the amount of calculation can be reduced, which helps reduce the calculation load.
For example, this application provides the following items:
(Item 1)
A three-dimensional reconstruction method,
acquiring a plurality of frames of images to be processed obtained by the imaging device scanning the target to be reconstructed;
determining a target pixel point and its camera pose parameters for each frame of the pending image belonging to the reconstruction target using each frame of the pending image and a calibration parameter of the imaging device;
Sequentially dividing the image data of each frame of the image to be processed into corresponding data sets according to a predetermined division policy, the image data including at least the target pixel point;
determining pose optimization parameters for each said data set using image data of each said data set and image data and pose optimization parameters of a data set before which the time series is placed in sequence;
adjusting camera pose parameters of the image to be processed to which image data included in the dataset belongs, using pose optimization parameters of each dataset;
Using a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the image to be processed, a reconstruction process is executed on the image data of the image to be processed, and the three-dimensional image of the target to be reconstructed is A three-dimensional reconstruction method, comprising: obtaining a model.
(Item 2)
Determining pose optimization parameters for each of the datasets by sequentially using the image data of each of the datasets and the image data and pose optimization parameters of a dataset before which the time series is arranged. ,
using each said data set in turn as a current data set and selecting at least one data set whose time series is placed before said current data set as a candidate data set;
determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset;
obtaining pose optimization parameters for the current dataset using at least pose optimization parameters of the candidate dataset and spatial transformation parameters between the current dataset and the candidate dataset; updating pose optimization parameters of the set;
The three-dimensional reconstruction method described in item 1.
(Item 3)
obtaining pose optimization parameters for the current dataset using the pose optimization parameters of the at least the candidate dataset and spatial transformation parameters between the current dataset and the candidate dataset; Updating pose optimization parameters for a dataset is
using as a dataset pair two datasets corresponding to each spatial transformation parameter respectively associated with the current dataset and the candidate dataset before which the time series is placed;
constructing an objective function for the pose optimization parameters using the spatial transformation parameters of each of the dataset pairs and the respective pose optimization parameters;
Solving the objective function by a predetermined solution method to obtain pose optimization parameters for datasets included in the current dataset and datasets corresponding to each of the candidate datasets before which the time series is placed. to do and include
The three-dimensional reconstruction method described in item 2.
(Item 4)
Determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset,
searching for a group of matching-waiting image data that satisfies a predetermined matching condition within the candidate data set and the current data set;
Obtaining matching pixel point pairs between the matching image data of each group based on predetermined image features extracted from the matching image data of each group;
Among the matching pixel point pairs, pixel points belonging to the current dataset are mapped to a three-dimensional space to obtain a first three-dimensional matching point, and among the matching pixel point pairs, pixel points belonging to the candidate dataset are mapped. mapping pixel points to the three-dimensional space to obtain second three-dimensional matching points;
aligning the first three-dimensional matching point and the second three-dimensional matching point to obtain the spatial transformation parameter;
The three-dimensional reconstruction method described in item 2.
(Item 5)
Aligning the first three-dimensional matching point and the second three-dimensional matching point to obtain the spatial transformation parameter,
obtaining a first pose transformation parameter between the first three-dimensional matching point and the second three-dimensional matching point;
Pose optimization is performed on the first three-dimensional matching point using the first pose transformation parameter and a predetermined pose transformation parameter, and the first optimized matching point and the second optimized matching point are and obtaining matching points, respectively.
The degree of coincidence between the second three-dimensional matching point and the first optimization matching point and the second optimization matching point is calculated, and the pose used by the optimization matching point with a high degree of coincidence is calculated. selecting the transformation parameter as a second pose transformation parameter;
The first three-dimensional matching point and the second three-dimensional matching point are aligned using a predetermined alignment method using the second pose transformation parameter as an initial value, and the current data set and the candidate data are aligned. retrieving the spatial transformation parameters with the set;
The three-dimensional reconstruction method described in item 4.
(Item 6)
After determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset, and at least a pose optimization of the candidate dataset. Before obtaining pose optimization parameters of the current dataset using a transformation parameter and a spatial transformation parameter of the current dataset and the candidate dataset, the three-dimensional reconstruction method comprises:
further comprising selecting a spatial transformation parameter that satisfies a predetermined parameter screening condition from spatial transformation parameters of the current data set and each of the candidate data sets;
The predetermined parameter screening condition includes: the candidate dataset associated with the spatial transformation parameter is adjacent to the current dataset, and the spatial transformation parameter is used to pose the first three-dimensional matching point. The matching degree between the optimized matching point obtained by the above-mentioned second three-dimensional matching point and the second three-dimensional matching point is greater than a predetermined matching degree threshold.
The three-dimensional reconstruction method described in item 4.
(Item 7)
Selecting at least one dataset whose time series is placed before the current dataset as a candidate dataset, comprising:
building a Bag of Words model using predetermined image features of the image data of the current dataset and the dataset before which the time series is placed;
selecting image data of an image to be processed that is located in a predetermined time series in the current data set as image data to be matched;
querying a predetermined range of the Bag of Words model for a predetermined image feature whose similarity score with the predetermined image feature of the image data awaiting matching is greater than a predetermined similarity threshold;
using a dataset in which the image data to which the queried predetermined image feature belongs is located, and a dataset adjacent to the current dataset as the candidate dataset;
The predetermined range includes predetermined image features of image data to which a dataset to which it belongs is not adjacent to the current dataset and is not included in the current dataset.
The three-dimensional reconstruction method described in item 2.
(Item 8)
before querying a predetermined image feature from a predetermined range of the Bag of Words model, the predetermined image feature having a similarity score with the predetermined image feature of the image data waiting for matching is greater than a predetermined similarity threshold; teeth,
obtaining a maximum score value of similarity scores between each of the image data and the matching waiting image data in the current data set and adjacent data sets;
further comprising using one of a predetermined multiple of the maximum score value and a predetermined score value as a predetermined similarity threshold.
The three-dimensional reconstruction method described in item 7.
(Item 9)
Sequentially dividing the image data of each frame of the image to be processed into corresponding data sets according to the predetermined division policy,
sequentially using each frame of the image to be processed as the current image to be processed;
When dividing the image data of the current image to be processed, when the last data set of the existing data set satisfies a predetermined overflow condition, the image data of the image to be processed of the latest frames in the last data set is divided. acquiring and storing data in the newly created dataset as a new final dataset, and dividing image data of the current pending image into the new final dataset.
The three-dimensional reconstruction method described in item 1.
(Item 10)
The predetermined overflow condition is:
The number of frames of the image to be processed corresponding to the image data included in the last data set is greater than or equal to a predetermined frame number threshold, and the image to be processed to which any of the image data in the last data set belongs and the camera position of the current image to be processed is greater than a predetermined distance threshold, and the camera orientation angle of the image to be processed to which any of the image data of the last data set belongs and the current The difference from the camera orientation angle of the image to be processed is greater than a predetermined angle threshold;
The camera position and camera orientation angle are calculated using camera pose parameters of the image to be processed.
The three-dimensional reconstruction method described in item 9.
(Item 11)
Each frame of the image to be processed includes color data and depth data, and each frame of the image to be processed and the calibration parameters of the imaging device are used to calculate the image of the image to be processed that belongs to the target to be reconstructed. Determining the target pixel point for each frame is
obtaining an included angle between the normal vector of each pixel point included in the depth data after alignment with the color data and the gravitational direction of the image to be processed;
Projecting each of the pixel points in the three-dimensional space in the direction of gravity to obtain a height value of each of the pixel points in the three-dimensional space;
analyzing a height value of a pixel point whose included angle satisfies a predetermined angle condition to obtain a planar height of the target awaiting reconstruction;
using the planar height to screen target pixel points belonging to the object to be reconstructed in the color data;
The three-dimensional reconstruction method according to any one of items 1-10.
(Item 12)
Analyzing the height value of a pixel point whose included angle satisfies a predetermined angle condition to obtain the planar height of the target awaiting reconstruction,
using the height values of the pixel points whose included angle satisfies a predetermined angle condition as a height set;
performing a cluster analysis on the height values in the height set to obtain a planar height of the reconfiguration target.
The three-dimensional reconstruction method described in item 11.
(Item 13)
After determining pose optimization parameters for each of the datasets using sequentially the image data of each of the datasets and the image data and pose optimization parameters of the dataset whose time series is placed before it; The three-dimensional reconstruction method includes:
sequentially mapping the image data of each of the datasets into a three-dimensional space to obtain a three-dimensional point cloud corresponding to each of the datasets;
further comprising using the pose optimization parameters of each of the datasets to adjust the corresponding three-dimensional point cloud.
The three-dimensional reconstruction method according to any one of items 1-12.
(Item 14)
An interaction method based on three-dimensional reconstruction, comprising:
Obtaining a three-dimensional model of a target awaiting reconstruction, the three-dimensional model being obtained using the three-dimensional reconstruction method described in any one of items 1-13. and,
constructing a three-dimensional map of the scene in which the photographing device is located using a predetermined visual-inertial navigation method, and obtaining current pose information of the photographing device in the three-dimensional map;
Displaying the three-dimensional model in a scene image currently being captured by the imaging device based on the pose information, the interaction method based on the three-dimensional reconstruction.
(Item 15)
A measurement method based on three-dimensional reconstruction, comprising:
Obtaining a three-dimensional model of a target awaiting reconstruction, the three-dimensional model being obtained using the three-dimensional reconstruction method described in any one of items 1-13. and,
receiving multiple measurement points set by the user in the 3D model;
and obtaining distances between the plurality of measurement points to obtain distances between positions corresponding to the plurality of measurement points on the target awaiting reconstruction. Measuring method.
(Item 16)
A three-dimensional reconstruction device,
an image acquisition unit configured to acquire a plurality of frames of pending images obtained by scanning the reconstruction target with the imaging device;
configured to use each frame of the pending image and a calibration parameter of the imaging device to determine a target pixel point of each frame of the pending image belonging to the reconstruction target and its camera pose parameters; a first deciding section;
A data division unit configured to sequentially divide image data of each frame of the image to be processed into corresponding data sets according to a predetermined division policy, wherein the image data includes at least data including the target pixel point. A dividing part,
configured to sequentially use the image data of each said dataset and the image data and pose optimization parameters of the dataset before which the time series is placed to determine pose optimization parameters for each said dataset; a second determining unit,
a parameter adjustment unit configured to use pose optimization parameters of each of the datasets to adjust camera pose parameters of a pending image to which image data included in the dataset belongs;
Using a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the image to be processed, a reconstruction process is executed on the image data of the image to be processed, and the three-dimensional image of the target to be reconstructed is The three-dimensional reconstruction device, comprising: a model reconstruction unit configured to obtain a model.
(Item 17)
The second determining unit is
a dataset selection sub configured to sequentially use each said dataset as a current dataset and select at least one dataset whose time series is placed before said current dataset as a candidate dataset; Department and
a spatial transformation parameter sub-unit configured to determine spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset; ,
obtaining pose optimization parameters for the current dataset using at least pose optimization parameters of the candidate dataset and spatial transformation parameters between the current dataset and the candidate dataset; a pose optimization parameter sub-portion configured to update pose optimization parameters of the set;
The three-dimensional reconstruction device according to item 16.
(Item 18)
The pose optimization parameter sub-section includes:
a dataset configured to use as a dataset pair two datasets corresponding to each spatial transformation parameter respectively associated with the current dataset and the candidate dataset before which the time series is placed; Pair part and
an objective function construction unit configured to construct an objective function regarding the pose optimization parameters using the spatial transformation parameters of each of the dataset pairs and the respective pose optimization parameters;
Solving the objective function by a predetermined solution method to obtain pose optimization parameters for datasets included in the current dataset and datasets corresponding to each of the candidate datasets before which the time series is placed. an objective function solver configured to
The three-dimensional reconstruction device according to item 17.
(Item 19)
The spatial transformation parameter sub-section is
an image data search unit configured to search for one group of matching-waiting image data that satisfies a predetermined matching condition within the candidate data set and the current data set;
a matching pixel point selection unit configured to obtain matching pixel point pairs between the matching image data of each group based on predetermined image features extracted from the matching image data of each group;
Among the matching pixel point pairs, pixel points belonging to the current dataset are mapped to a three-dimensional space to obtain a first three-dimensional matching point, and among the matching pixel point pairs, pixel points belonging to the candidate dataset are mapped. a three-dimensional space mapping unit configured to map pixel points onto the three-dimensional space to obtain second three-dimensional matching points;
a three-dimensional matching point alignment unit configured to align the first three-dimensional matching point and the second three-dimensional matching point to obtain the spatial transformation parameter.
The three-dimensional reconstruction device according to item 17.
(Item 20)
The three-dimensional matching point alignment section is
a first pose transformation parameter sub-unit configured to obtain a first pose transformation parameter between the first three-dimensional matching point and the second three-dimensional matching point;
Pose optimization is performed on the first three-dimensional matching point using the first pose transformation parameter and a predetermined pose transformation parameter, and the first optimized matching point and the second optimized matching point are a three-dimensional matching point optimization sub-unit configured to obtain the matching points, respectively;
The degree of coincidence between the second three-dimensional matching point and the first optimization matching point and the second optimization matching point is calculated, and the pose used by the optimization matching point with a high degree of coincidence is calculated. a second pose transformation parameter sub-section configured to select the transformation parameter as a second pose transformation parameter;
The first three-dimensional matching point and the second three-dimensional matching point are aligned using a predetermined alignment method using the second pose transformation parameter as an initial value, and the current data set and the candidate are aligned. a spatial transformation parameter sub-portion configured to obtain spatial transformation parameters with the dataset;
The three-dimensional reconstruction device according to item 19.
(Item 21)
The spatial transformation parameter sub-section is
After determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset, and at least a pose optimization of the candidate dataset. the current dataset and each candidate data set before obtaining pose optimization parameters for the current dataset using spatial transformation parameters between the current dataset and the candidate dataset. further comprising a transformation parameter screening unit configured to select from the set of spatial transformation parameters a spatial transformation parameter that satisfies a predetermined parameter screening condition;
The predetermined parameter screening condition includes: the candidate dataset associated with the spatial transformation parameter is adjacent to the current dataset, and the spatial transformation parameter is used to pose the first three-dimensional matching point. The matching degree between the optimized matching point obtained by the above-mentioned second three-dimensional matching point and the second three-dimensional matching point is greater than a predetermined matching degree threshold.
The three-dimensional reconstruction device according to item 19.
(Item 22)
The data set selection sub-unit includes:
a Bag of Words model builder configured to build a Bag of Words model using predetermined image features of the image data of the current dataset and the dataset before which the time series is placed; ,
a matching pending image data unit configured to select image data located in a predetermined time series of the current dataset of pending processing images to which it belongs as matching pending image data;
an image feature query unit configured to query, from a predetermined range of the Bag of Words model, a predetermined image feature whose similarity score with a predetermined image feature of the image data awaiting matching is greater than a predetermined similarity threshold; and,
a dataset in which the image data to which the queried predetermined image feature belongs is located, and a candidate dataset unit configured to use a dataset adjacent to the current dataset as the candidate dataset;
The predetermined range includes predetermined image features of image data to which a dataset to which it belongs is not adjacent to the current dataset and is not included in the current dataset.
The three-dimensional reconstruction device according to item 16.
(Item 23)
The data set selection sub-unit includes:
From the predetermined range of the Bag of Words model, before querying for a predetermined image feature whose similarity score with the predetermined image feature of the image data to be matched is greater than a predetermined similarity threshold, a maximum similarity score value obtaining unit configured to obtain a maximum score value of similarity scores between each of the image data and the matching-waiting image data in adjacent data sets;
further comprising a predetermined similarity threshold determination unit configured to use either one of the predetermined multiple of the maximum score value and the predetermined score value as a predetermined similarity threshold.
The three-dimensional reconstruction device according to item 22.
(Item 24)
The data division section is
a current pending image determining sub-unit configured to sequentially use each frame of the pending image as a current pending image;
When dividing the image data of the current image to be processed, when the last data set of the existing data set satisfies a predetermined overflow condition, the image data of the image to be processed of the latest frames in the last data set is divided. a data processing subsystem configured to obtain and store data in the newly created dataset as a new final dataset, and to divide image data of the current pending image into a new final dataset; comprising a section and
The three-dimensional reconstruction device according to item 16.
(Item 25)
The predetermined overflow condition is:
The number of frames of the image to be processed corresponding to the image data included in the last data set is greater than or equal to a predetermined frame number threshold, and the image to be processed to which any of the image data in the last data set belongs and the camera position of the current image to be processed is greater than a predetermined distance threshold, and the camera orientation angle of the image to be processed to which any of the image data of the last data set belongs and the current The difference from the camera orientation angle of the image to be processed is greater than a predetermined angle threshold;
The camera position and camera orientation angle are calculated using camera pose parameters of the image to be processed.
The three-dimensional reconstruction device according to item 24.
(Item 26)
Each frame of the image to be processed includes color data and depth data, and the first determining unit:
an included angle acquisition sub-unit configured to obtain an included angle between a normal vector of each pixel point included in the depth data after alignment with the color data and a gravitational direction of the image to be processed;
a height acquisition sub-unit configured to project each pixel point in a three-dimensional space in the direction of gravity and obtain a height value of each pixel point in the three-dimensional space;
a height analysis sub-unit configured to analyze a height value of a pixel point whose included angle satisfies a predetermined angle condition to obtain a planar height of the target to be reconstructed;
a pixel screening sub-unit configured to screen target pixel points belonging to the object to be reconstructed in the color data according to the planar height;
The three-dimensional reconstruction device according to any one of items 16-26.
(Item 27)
The height analysis sub-section is
a height set acquisition unit configured to use, as a height set, a height value of the pixel point whose included angle satisfies a predetermined angle condition;
a height cluster analysis unit configured to perform cluster analysis on the height values in the height set to obtain a planar height of the reconfiguration target.
The three-dimensional reconstruction device according to item 26.
(Item 28)
The three-dimensional reconstruction device includes:
a three-dimensional mapping unit configured to sequentially map image data of each of the datasets onto a three-dimensional space to obtain a three-dimensional point cloud corresponding to each of the datasets;
further comprising a point cloud adjustment unit configured to use the pose optimization parameters of each of the data sets to adjust the corresponding three-dimensional point cloud.
The three-dimensional reconstruction device according to any one of items 16-27.
(Item 29)
An interaction device based on three-dimensional reconstruction,
a model acquisition unit configured to acquire a three-dimensional model of a target awaiting reconstruction, wherein the three-dimensional model is obtained by the three-dimensional reconstruction device according to item 16;
a mapping and positioning unit configured to construct a three-dimensional map of the scene in which the photographing device is placed according to a predetermined visual-inertial navigation scheme and obtain current pose information of the photographing device in the three-dimensional map;
an interaction device based on three-dimensional reconstruction, comprising: a display interaction unit configured to display the three-dimensional model on a scene image currently being photographed by the photographing device based on the pose information.
(Item 30)
A measurement device based on three-dimensional reconstruction, comprising:
a model acquisition unit configured to acquire a three-dimensional model of a target awaiting reconstruction, wherein the three-dimensional model is obtained by the three-dimensional reconstruction device according to item 16;
a display interaction unit configured to receive a plurality of measurement points set by a user on the three-dimensional model;
a distance acquisition unit configured to obtain distances between the plurality of measurement points and obtain distances between positions corresponding to the plurality of measurement points on the target awaiting reconstruction; Measurement device based on three-dimensional reconstruction.
(Item 31)
An electronic device,
comprising a memory and a processor coupled to each other, the processor realizing the three-dimensional reconstruction method according to any one of items 1-13, or implementing the three-dimensional reconstruction-based interaction method according to item 14. Electronic equipment configured to execute program instructions stored in said memory in order to realize or realize the three-dimensional reconstruction-based measurement method according to item 15.
(Item 32)
A computer-readable storage medium storing program instructions, the program instructions, when executed by a processor, realizing the three-dimensional reconstruction method according to any one of items 1-13, or implementing the three-dimensional reconstruction method according to item 14. A computer-readable storage medium that implements the interaction method based on three-dimensional reconstruction according to item 15 or the measurement method based on three-dimensional reconstruction according to item 15.
(Item 33)
comprising a computer readable code, said computer readable code being operative in an electronic device and implementing a three-dimensional reconstruction method according to any one of items 1-13, when executed by a processor of said electronic device, or A computer program that implements the interaction method based on three-dimensional reconstruction described in item 14 or the measurement method based on three-dimensional reconstruction described in item 15.
(Item 34)
When operating on a computer, cause the computer to execute the three-dimensional reconstruction method described in any one of items 1-13, or to execute the interaction method based on three-dimensional reconstruction described in item 14, or to cause the computer to execute the interaction method based on three-dimensional reconstruction described in item 15. A computer program product that causes the three-dimensional reconstruction-based measurement method described in .

Claims (34)

A three-dimensional reconstruction method,
acquiring a plurality of frames of images to be processed obtained by the imaging device scanning the target to be reconstructed;
determining a target pixel point and its camera pose parameters for each frame of the pending image belonging to the reconstruction target using each frame of the pending image and a calibration parameter of the imaging device;
Sequentially dividing the image data of each frame of the image to be processed into corresponding data sets according to a predetermined division policy, the image data including at least the target pixel point;
determining pose optimization parameters for each said data set using image data of each said data set and image data and pose optimization parameters of a data set before which the time series is placed in sequence;
adjusting camera pose parameters of the image to be processed to which image data included in the dataset belongs, using pose optimization parameters of each dataset;
Using a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the image to be processed, a reconstruction process is executed on the image data of the image to be processed, and the three-dimensional image of the target to be reconstructed is A three-dimensional reconstruction method, comprising: obtaining a model. Determining pose optimization parameters for each of the datasets by sequentially using the image data of each of the datasets and the image data and pose optimization parameters of a dataset before which the time series is arranged. ,
using each said data set in turn as a current data set and selecting at least one data set whose time series is placed before said current data set as a candidate data set;
determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset;
obtaining pose optimization parameters for the current dataset using at least pose optimization parameters of the candidate dataset and spatial transformation parameters between the current dataset and the candidate dataset; The three-dimensional reconstruction method of claim 1, comprising: updating a set of pose optimization parameters. obtaining pose optimization parameters for the current dataset using the pose optimization parameters of the at least the candidate dataset and spatial transformation parameters between the current dataset and the candidate dataset; Updating pose optimization parameters for a dataset is
using as a dataset pair two datasets corresponding to each spatial transformation parameter respectively associated with the current dataset and the candidate dataset before which the time series is placed;
constructing an objective function for the pose optimization parameters using the spatial transformation parameters of each of the dataset pairs and the respective pose optimization parameters;
Solving the objective function by a predetermined solution method to obtain pose optimization parameters for datasets included in the current dataset and datasets corresponding to each of the candidate datasets before which the time series is placed. The three-dimensional reconstruction method according to claim 2, comprising: Determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset,
searching for a group of matching-waiting image data that satisfies a predetermined matching condition within the candidate data set and the current data set;
Obtaining matching pixel point pairs between the matching image data of each group based on predetermined image features extracted from the matching image data of each group;
Among the matching pixel point pairs, pixel points belonging to the current dataset are mapped to a three-dimensional space to obtain a first three-dimensional matching point, and among the matching pixel point pairs, pixel points belonging to the candidate dataset are mapped. mapping pixel points to the three-dimensional space to obtain second three-dimensional matching points;
The three-dimensional reconstruction method according to claim 2, comprising: aligning the first three-dimensional matching point and the second three-dimensional matching point to obtain the spatial transformation parameter. Aligning the first three-dimensional matching point and the second three-dimensional matching point to obtain the spatial transformation parameter,
obtaining a first pose transformation parameter between the first three-dimensional matching point and the second three-dimensional matching point;
Pose optimization is performed on the first three-dimensional matching point using the first pose transformation parameter and a predetermined pose transformation parameter, and the first optimized matching point and the second optimized matching point are and obtaining matching points, respectively.
The degree of coincidence between the second three-dimensional matching point and the first optimization matching point and the second optimization matching point is calculated, and the pose used by the optimization matching point with the high degree of coincidence is calculated. selecting the transformation parameter as a second pose transformation parameter;
The first three-dimensional matching point and the second three-dimensional matching point are aligned using a predetermined alignment method using the second pose transformation parameter as an initial value, and the current data set and the candidate data are aligned. 5. The three-dimensional reconstruction method according to claim 4, comprising: obtaining a spatial transformation parameter with a set. After determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset, and at least a pose optimization of the candidate dataset. Before obtaining pose optimization parameters of the current dataset using a transformation parameter and a spatial transformation parameter of the current dataset and the candidate dataset, the three-dimensional reconstruction method comprises:
further comprising selecting a spatial transformation parameter that satisfies a predetermined parameter screening condition from spatial transformation parameters of the current data set and each of the candidate data sets;
The predetermined parameter screening condition includes: the candidate dataset associated with the spatial transformation parameter is adjacent to the current dataset, and the spatial transformation parameter is used to pose the first three-dimensional matching point. 5. The three-dimensional reconstruction according to claim 4, wherein the degree of coincidence between the optimized matching point obtained by How to configure. Selecting at least one dataset whose time series is placed before the current dataset as a candidate dataset, comprising:
building a Bag of Words model using predetermined image features of the image data of the current dataset and the dataset before which the time series is placed;
selecting image data of an image to be processed that is located in a predetermined time series in the current data set as image data to be matched;
querying a predetermined image feature from a predetermined range of the Bag of Words model that has a similarity score with a predetermined image feature of the image data awaiting matching that is greater than a predetermined similarity threshold;
using a dataset in which the image data to which the queried predetermined image feature belongs is located, and a dataset adjacent to the current dataset as the candidate dataset;
The three-dimensional reconstruction method according to claim 2, wherein the predetermined range includes predetermined image features of image data to which a dataset to which it belongs is not adjacent to the current dataset and is not included in the current dataset. . Before querying a predetermined image feature from a predetermined range of the Bag of Words model, the predetermined image feature having a similarity score with the predetermined image feature of the image data waiting for matching is greater than a predetermined similarity threshold. teeth,
obtaining a maximum score value of similarity scores between each of the image data and the matching waiting image data in the current data set and adjacent data sets;
The three-dimensional reconstruction method according to claim 7, further comprising: using any one of a predetermined multiple of the maximum score value and a predetermined score value as a predetermined similarity threshold. Sequentially dividing the image data of each frame of the image to be processed into corresponding data sets according to the predetermined division policy,
sequentially using each frame of the image to be processed as a current image to be processed;
When dividing the image data of the current image to be processed, when the last data set of the existing data set satisfies a predetermined overflow condition, the image data of the image to be processed of the latest frames in the last data set is divided. acquiring and storing data in the newly created dataset as a new final dataset, and dividing the image data of the current pending image into the new final dataset. The three-dimensional reconstruction method described in . The predetermined overflow condition is:
The number of frames of the image to be processed corresponding to the image data included in the last data set is greater than or equal to a predetermined frame number threshold, and the image to be processed to which any of the image data in the last data set belongs and the camera position of the current image to be processed is greater than a predetermined distance threshold, and the camera orientation angle of the image to be processed to which any of the image data of the last dataset belongs and the current image The difference from the camera orientation angle of the image to be processed is greater than a predetermined angle threshold;
The three-dimensional reconstruction method according to claim 9, wherein the camera position and the camera orientation angle are calculated using camera pose parameters of the image to be processed. Each frame of the image to be processed includes color data and depth data, and each frame of the image to be processed and the calibration parameters of the imaging device are used to calculate the image of the image to be processed that belongs to the target to be reconstructed. Determining the target pixel point for each frame is
obtaining an included angle between the normal vector of each pixel point included in the depth data after alignment with the color data and the gravitational direction of the image to be processed;
Projecting each of the pixel points in the three-dimensional space in the direction of gravity to obtain a height value of each of the pixel points in the three-dimensional space;
analyzing a height value of a pixel point whose included angle satisfies a predetermined angle condition to obtain a planar height of the target awaiting reconstruction;
11. The three-dimensional reconstruction method according to claim 1, further comprising screening target pixel points belonging to the object to be reconstructed in the color data using the planar height. Analyzing the height value of a pixel point whose included angle satisfies a predetermined angle condition to obtain the planar height of the target awaiting reconstruction,
using the height values of the pixel points whose included angle satisfies a predetermined angle condition as a height set;
12. The three-dimensional reconstruction method according to claim 11, comprising: performing cluster analysis on the height values in the height set to obtain a planar height of the target to be reconstructed. After determining pose optimization parameters for each of the datasets using sequentially the image data of each of the datasets and the image data and pose optimization parameters of the dataset whose time series is placed before it; The three-dimensional reconstruction method includes:
sequentially mapping the image data of each of the datasets into a three-dimensional space to obtain a three-dimensional point cloud corresponding to each of the datasets;
13. A three-dimensional reconstruction method according to any one of claims 1-12, further comprising: using the pose optimization parameters of each dataset to adjust the corresponding three-dimensional point cloud. . An interaction method based on three-dimensional reconstruction, comprising:
Obtaining a three-dimensional model of a target awaiting reconstruction, the three-dimensional model being obtained using the three-dimensional reconstruction method according to any one of claims 1 to 13. And,
constructing a three-dimensional map of the scene in which the photographing device is located using a predetermined visual-inertial navigation method, and obtaining current pose information of the photographing device in the three-dimensional map;
Displaying the three-dimensional model in a scene image currently being captured by the imaging device based on the pose information, the interaction method based on the three-dimensional reconstruction. A measurement method based on three-dimensional reconstruction, comprising:
Obtaining a three-dimensional model of a target awaiting reconstruction, the three-dimensional model being obtained using the three-dimensional reconstruction method according to any one of claims 1 to 13. And,
receiving multiple measurement points set by the user in the 3D model;
and obtaining distances between the plurality of measurement points to obtain distances between positions corresponding to the plurality of measurement points on the target awaiting reconstruction. Measuring method. A three-dimensional reconstruction device,
an image acquisition unit configured to acquire a plurality of frames of pending images obtained by scanning the reconstruction target with the imaging device;
configured to use each frame of the pending image and a calibration parameter of the imaging device to determine a target pixel point of each frame of the pending image belonging to the reconstruction target and its camera pose parameters; a first decision section;
A data division unit configured to sequentially divide image data of each frame of the image to be processed into corresponding data sets according to a predetermined division policy, wherein the image data includes at least data including the target pixel point. A dividing part,
configured to sequentially use the image data of each said dataset and the image data and pose optimization parameters of the dataset before which the time series is placed to determine pose optimization parameters for each said dataset; a second determining unit,
a parameter adjustment unit configured to use pose optimization parameters of each of the datasets to adjust camera pose parameters of a pending image to which image data included in the dataset belongs;
Using a predetermined three-dimensional reconstruction method and the adjusted camera pose parameters of the image to be processed, a reconstruction process is executed on the image data of the image to be processed, and the three-dimensional image of the target to be reconstructed is The three-dimensional reconstruction device, comprising: a model reconstruction unit configured to obtain a model. The second determining unit includes:
a dataset selection sub configured to sequentially use each said dataset as a current dataset and select at least one dataset whose time series is placed before said current dataset as a candidate dataset; Department and
a spatial transformation parameter sub-unit configured to determine spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset; ,
obtaining pose optimization parameters for the current dataset using at least pose optimization parameters of the candidate dataset and spatial transformation parameters between the current dataset and the candidate dataset; 17. The three-dimensional reconstruction device according to claim 16, comprising a pose optimization parameter sub-unit configured to update a set of pose optimization parameters. The pose optimization parameter sub-section includes:
a dataset configured to use as a dataset pair two datasets corresponding to each spatial transformation parameter respectively associated with the current dataset and the candidate dataset before which the time series is placed; Pair part and
an objective function construction unit configured to construct an objective function regarding the pose optimization parameters using the spatial transformation parameters of each of the dataset pairs and the respective pose optimization parameters;
Solving the objective function by a predetermined solution method to obtain pose optimization parameters for datasets included in the current dataset and datasets corresponding to each of the candidate datasets before which the time series is placed. The three-dimensional reconstruction device according to claim 17, further comprising: an objective function solving unit configured to perform. The spatial transformation parameter sub-section is
an image data search unit configured to search for one group of matching-waiting image data that satisfies a predetermined matching condition within the candidate data set and the current data set;
a matching pixel point selection unit configured to obtain matching pixel point pairs between the matching image data of each group based on predetermined image features extracted from the matching image data of each group;
Among the matching pixel point pairs, pixel points belonging to the current dataset are mapped to a three-dimensional space to obtain a first three-dimensional matching point, and among the matching pixel point pairs, pixel points belonging to the candidate dataset are mapped. a three-dimensional space mapping unit configured to map pixel points onto the three-dimensional space to obtain second three-dimensional matching points;
17. A three-dimensional matching point alignment unit configured to align the first three-dimensional matching point and the second three-dimensional matching point to obtain the spatial transformation parameter. The three-dimensional reconstruction device described in . The three-dimensional matching point alignment section is
a first pose transformation parameter sub-unit configured to obtain a first pose transformation parameter between the first three-dimensional matching point and the second three-dimensional matching point;
Pose optimization is performed on the first three-dimensional matching point using the first pose transformation parameter and a predetermined pose transformation parameter, and the first optimized matching point and the second optimized matching point are a three-dimensional matching point optimization sub-unit configured to obtain the matching points, respectively;
The degree of coincidence between the second three-dimensional matching point and the first optimization matching point and the second optimization matching point is calculated, and the pose used by the optimization matching point with a high degree of coincidence is calculated. a second pose transformation parameter sub-section configured to select the transformation parameter as a second pose transformation parameter;
The first three-dimensional matching point and the second three-dimensional matching point are aligned using a predetermined alignment method using the second pose transformation parameter as an initial value, and the current data set and the candidate are aligned. 20. The three-dimensional reconstruction device according to claim 19, comprising: a spatial transformation parameter sub-unit configured to obtain spatial transformation parameters with the data set. The spatial transformation parameter sub-section is
After determining spatial transformation parameters between the current dataset and the candidate dataset using image data of the current dataset and image data of the candidate dataset, and at least a pose optimization of the candidate dataset. the current dataset and each candidate data set before obtaining pose optimization parameters for the current dataset using spatial transformation parameters between the current dataset and the candidate dataset. further comprising a transformation parameter screening unit configured to select from the set of spatial transformation parameters a spatial transformation parameter that satisfies a predetermined parameter screening condition;
The predetermined parameter screening condition includes: the candidate dataset associated with the spatial transformation parameter is adjacent to the current dataset, and the spatial transformation parameter is used to pose the first three-dimensional matching point. 20. The three-dimensional reconstruction according to claim 19, wherein the degree of coincidence between the optimized matching point obtained by Component device. The data set selection sub-unit includes:
a Bag of Words model builder configured to build a Bag of Words model using predetermined image features of the image data of the current dataset and the dataset before which the time series is placed; ,
a matching pending image data unit configured to select image data located in a predetermined time series of the current dataset of pending processing images to which it belongs as matching pending image data;
an image feature query unit configured to query, from a predetermined range of the Bag of Words model, a predetermined image feature whose similarity score with a predetermined image feature of the image data awaiting matching is greater than a predetermined similarity threshold; and,
a dataset in which the image data to which the queried predetermined image feature belongs is located, and a candidate dataset unit configured to use a dataset adjacent to the current dataset as the candidate dataset;
The three-dimensional reconstruction device according to claim 16, wherein the predetermined range includes predetermined image features of image data to which a dataset to which it belongs is not adjacent to the current dataset and is not included in the current dataset. . The data set selection sub-unit includes:
From the predetermined range of the Bag of Words model, before querying for a predetermined image feature whose similarity score with the predetermined image feature of the image data to be matched is greater than a predetermined similarity threshold, a maximum similarity score value obtaining unit configured to obtain a maximum score value of similarity scores between each of the image data and the matching-waiting image data in adjacent data sets;
23. A predetermined similarity threshold determination unit configured to use any one of the predetermined multiple of the maximum score value and the predetermined score value as a predetermined similarity threshold. 3D reconstruction device. The data division section is
a current pending image determining sub-unit configured to sequentially use each frame of the pending image as a current pending image;
When dividing the image data of the current image to be processed, when the last data set of the existing data set satisfies a predetermined overflow condition, the image data of the image to be processed of the latest frames in the last data set is divided. a data processing subsystem configured to obtain and store data in the newly created dataset as a new final dataset, and to divide image data of the current pending image into a new final dataset; The three-dimensional reconstruction device according to claim 16, comprising: a. The predetermined overflow condition is:
The number of frames of the image to be processed corresponding to the image data included in the last data set is greater than or equal to a predetermined frame number threshold, and the image to be processed to which any of the image data in the last data set belongs and the camera position of the current image to be processed is greater than a predetermined distance threshold, and the camera orientation angle of the image to be processed to which any of the image data of the last data set belongs and the current The difference from the camera orientation angle of the image to be processed is greater than a predetermined angle threshold;
The three-dimensional reconstruction device according to claim 24, wherein the camera position and the camera orientation angle are calculated using camera pose parameters of the image to be processed. Each frame of the image to be processed includes color data and depth data, and the first determining unit:
an included angle acquisition sub-unit configured to obtain an included angle between a normal vector of each pixel point included in the depth data after alignment with the color data and a gravitational direction of the image to be processed;
a height acquisition sub-unit configured to project each pixel point in a three-dimensional space in the direction of gravity and obtain a height value of each pixel point in the three-dimensional space;
a height analysis sub-unit configured to analyze a height value of a pixel point whose included angle satisfies a predetermined angle condition to obtain a planar height of the target to be reconstructed;
a pixel screening sub-unit configured to screen target pixel points belonging to the object to be reconstructed in the color data by the planar height. Reconfiguration device. The height analysis sub-section is
a height set acquisition unit configured to use, as a height set, a height value of the pixel point whose included angle satisfies a predetermined angle condition;
27. A height cluster analysis unit configured to perform cluster analysis on the height values in the height set to obtain a planar height of the target to be reconfigured. The three-dimensional reconstruction device described. The three-dimensional reconstruction device includes:
a three-dimensional mapping unit configured to sequentially map the image data of each of the datasets into a three-dimensional space to obtain a three-dimensional point cloud corresponding to each of the datasets;
28. A point cloud adjustment unit configured to use the pose optimization parameters of each data set to adjust the corresponding three-dimensional point cloud. The three-dimensional reconstruction device described in . An interaction device based on three-dimensional reconstruction,
a model acquisition unit configured to acquire a three-dimensional model of a target awaiting reconstruction, wherein the three-dimensional model is obtained by the three-dimensional reconstruction device according to claim 16;
a mapping and positioning unit configured to construct a three-dimensional map of the scene in which the photographing device is placed according to a predetermined visual-inertial navigation method and obtain current pose information of the photographing device in the three-dimensional map;
an interaction device based on three-dimensional reconstruction, comprising: a display interaction unit configured to display the three-dimensional model on a scene image currently being photographed by the photographing device based on the pose information. A measuring device based on three-dimensional reconstruction, comprising:
a model acquisition unit configured to acquire a three-dimensional model of a target awaiting reconstruction, wherein the three-dimensional model is obtained by the three-dimensional reconstruction device according to claim 16;
a display interaction unit configured to receive a plurality of measurement points set by a user on the three-dimensional model;
a distance acquisition unit configured to obtain distances between the plurality of measurement points and obtain distances between positions corresponding to the plurality of measurement points on the target awaiting reconstruction; Measurement device based on three-dimensional reconstruction. An electronic device,
comprising a memory and a processor coupled to each other, said processor implementing a three-dimensional reconstruction method according to any one of claims 1-13, or a three-dimensional reconstruction-based interaction according to claim 14. Electronic equipment configured to execute program instructions stored in said memory for implementing a method or for implementing a measurement method based on three-dimensional reconstruction according to claim 15. A computer-readable storage medium storing program instructions, which, when executed by a processor, implement a three-dimensional reconstruction method according to any one of claims 1-13, or claim 14. 16. A computer-readable storage medium for implementing the three-dimensional reconstruction-based interaction method as claimed in claim 15 or for implementing the three-dimensional reconstruction-based measurement method as claimed in claim 15. realizing a three-dimensional reconstruction method according to any one of claims 1 to 13, comprising a computer readable code, said computer readable code being operative in an electronic device and executed by a processor of said electronic device; Or a computer program for realizing the interaction method based on three-dimensional reconstruction according to claim 14, or for realizing the measuring method based on three-dimensional reconstruction according to claim 15. When operating on a computer, cause the computer to execute the three-dimensional reconstruction method according to any one of claims 1 to 13, or execute the interaction method based on three-dimensional reconstruction according to claim 14, or A computer program product for carrying out the three-dimensional reconstruction-based measurement method according to claim 15.

Images