CN115760588A

CN115760588A - Point cloud correction method and three-dimensional model generation method, device and equipment

Info

Publication number: CN115760588A
Application number: CN202211107264.8A
Authority: CN
Inventors: 王海君
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-09-09
Filing date: 2022-09-09
Publication date: 2023-03-07

Abstract

The disclosure provides a point cloud correction method, a three-dimensional model generation method, a device, equipment and a medium, and relates to the field of artificial intelligence, in particular to the technical fields of virtual reality, augmented reality, metas, computer vision, deep learning and the like. The point cloud correction method comprises the following steps: performing plane fitting according to the first point cloud data of the first object to obtain a plane model; correcting the first point cloud data according to the plane model to obtain corrected point cloud data; determining target point cloud data corresponding to a target boundary line in the target image in the corrected point cloud data, wherein the target boundary line is a boundary line between the first object and the second object; determining a depth value corresponding to a second pixel point according to the target point cloud data based on the position relationship between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to a second object in the target image; and correcting the second point cloud data of the second object according to the depth value.

Description

Point cloud correction method and three-dimensional model generation method, device and equipment

Technical Field

The disclosure relates to the field of artificial intelligence, in particular to the technical fields of virtual reality, augmented reality, meta universe, computer vision, deep learning and the like, and particularly relates to a point cloud correction method and a three-dimensional model generation method, device, equipment and medium.

Background

With the development of computer technology and network technology, various emerging technologies are in force. For example, three-dimensional reconstruction techniques that generate three-dimensional models of scenes have evolved rapidly. Generally, in a three-dimensional reconstruction technology, the accuracy requirement of point cloud data is high when a plane is generated, and the plane is distorted and the accuracy of a three-dimensional model is low due to inaccurate point cloud data.

Disclosure of Invention

The present disclosure is directed to a method, an apparatus, a device, and a medium for correcting a point cloud and a method, an apparatus, a device, and a medium for generating a three-dimensional model, so as to improve the accuracy of point cloud data and the three-dimensional model and avoid the situation of plane distortion.

According to an aspect of the present disclosure, there is provided a method of correcting a point cloud, including: performing plane fitting according to the first point cloud data of the first object to obtain a plane model representing the plane where the first object is located; correcting the first point cloud data according to the plane model to obtain corrected point cloud data; determining target point cloud data corresponding to a target boundary line in the target image in the corrected point cloud data, wherein the target boundary line is a boundary line between a first object and a second object in the target image; determining a depth value corresponding to a second pixel point according to the target point cloud data based on a position relation between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to a second object in the target image; and correcting the second point cloud data of the second object according to the depth value.

According to another aspect of the present disclosure, there is provided a method of generating a three-dimensional model, including: generating initial point cloud data corresponding to a target image according to the target image comprising a first object and a second object; the initial point cloud data comprises first point cloud data of a first object and second point cloud data of a second object; correcting the first point cloud data and the second point cloud data by adopting the point cloud correction method provided by the disclosure to obtain corrected point cloud data; and generating a three-dimensional model according to the corrected point cloud data.

According to another aspect of the present disclosure, there is provided a point cloud correction apparatus including: the plane fitting module is used for carrying out plane fitting according to the first point cloud data of the first object to obtain a plane model representing the plane where the first object is located; the first correction module is used for correcting the first point cloud data according to the plane model to obtain corrected point cloud data; the boundary point cloud determining module is used for determining target point cloud data corresponding to a target boundary line in the target image in the corrected point cloud data, wherein the target boundary line is a boundary line between a first object and a second object in the target image; the depth value determining module is used for determining a depth value corresponding to a second pixel point according to the target point cloud data based on the position relation between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to a second object in the target image; and the second correction module is used for correcting the second point cloud data of the second object according to the depth value.

According to another aspect of the present disclosure, there is provided a three-dimensional model generation apparatus including: the point cloud generating module is used for generating initial point cloud data corresponding to a target image according to the target image comprising a first object and a second object; the initial point cloud data comprises first point cloud data of a first object and second point cloud data of a second object; the point cloud correction module is used for correcting the first point cloud data and the second point cloud data by adopting the point cloud correction device provided by the disclosure to obtain corrected point cloud data; and the model generation module is used for generating a three-dimensional model according to the corrected point cloud data.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of correcting a point cloud and/or the method of generating a three-dimensional model provided by the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute a correction method of a point cloud and/or a generation method of a three-dimensional model provided by the present disclosure.

According to another aspect of the present disclosure, a computer program product is provided, comprising computer programs/instructions which, when executed by a processor, implement the method of correction of a point cloud and/or the method of generation of a three-dimensional model provided by the present disclosure.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a schematic view of an application scenario of a method and an apparatus for correcting a point cloud and generating a three-dimensional model according to an embodiment of the disclosure;

FIG. 2 is a schematic flow chart diagram of a method of correcting a point cloud according to an embodiment of the disclosure;

FIG. 3 is an implementation schematic diagram of a method of correcting a point cloud according to an embodiment of the disclosure;

FIG. 4 is a schematic diagram illustrating a principle of determining a depth value corresponding to a second pixel point according to an embodiment of the disclosure;

FIG. 5 is a flow chart diagram of a method of generating a three-dimensional model according to an embodiment of the present disclosure;

FIG. 6 is an effect diagram of a three-dimensional model generated by a method of generating a three-dimensional model according to an embodiment of the present disclosure;

fig. 7 is a block diagram of a structure of a correction apparatus of point cloud data according to an embodiment of the present disclosure;

fig. 8 is a block diagram of a structure of a three-dimensional model generation apparatus according to an embodiment of the present disclosure; and

fig. 9 is a block diagram of an electronic device for implementing a method of correcting point cloud data and/or a method of generating a three-dimensional model according to an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In three-dimensional reconstruction techniques, a three-dimensional model of an object is typically derived from point cloud data of the object generated based on an image. The generation of point cloud data relies on depth estimates obtained for a single image. In carrying out the disclosed concept, the inventors discovered: due to the influence of illumination or depth estimation precision and the like, the precision of the point cloud data is difficult to guarantee. Due to the limitation of the precision of the point cloud data, there are usually situations that the ground plane and/or the wall plane in the generated three-dimensional model are distorted, and a gap exists between the two joint planes.

In order to solve the technical problem, the present disclosure provides a method for correcting a point cloud and a method for generating a three-dimensional model, which improve the accuracy of a reconstructed three-dimensional model by correcting a point cloud of an object.

An application scenario of the method and apparatus provided by the present disclosure is described below with reference to fig. 1.

Fig. 1 is a schematic view of an application scenario of a method for correcting a point cloud and a method and an apparatus for generating a three-dimensional model according to an embodiment of the present disclosure.

As shown in fig. 1, the application scenario 100 of this embodiment may include an electronic device 110, and the electronic device 110 may be various electronic devices with processing functionality, including but not limited to a smartphone, a tablet, a laptop, a desktop computer, a server, and so on.

The electronic device 110 may process the input image 120, for example, specifically, generate point cloud data of an object according to the image 120, and then correct the point cloud data by using a plane fitting algorithm and the like to obtain corrected point cloud data.

In an embodiment, the object to be corrected may include, for example, two adjacent objects having a boundary line, and the electronic device may correct the point cloud data of one of the two objects first, and then correct the point cloud data of the other of the two objects according to a correction result, so as to ensure that the boundary line between the two objects does not have a gap in the three-dimensional model generated according to the corrected point cloud data, and both the two objects are planar structures.

In an embodiment, the electronic device 110 may further generate a three-dimensional model 130 of the scene depicted by the image 120 according to the corrected point cloud data. For example, the three-dimensional model 130 may be represented by corrected point cloud data.

In an embodiment, the application scenario 100 may further include a server 150. The electronic device 110 may be communicatively coupled to the server 150 via a network. The network may include wired or wireless communication links.

For example, the electronic device 110 may send the image 120 to the server 150, and the server 150 processes the image 120 to correct the point cloud data of the planar structure object in the image. The server 150 may, for example, send the corrected point cloud data to the electronic device 110 for the electronic device 110 to generate the three-dimensional model 130 from the corrected point cloud data. Alternatively, the image 120 may be processed, for example, by the electronic device 110, resulting in corrected point cloud data 140, and the electronic device 110 may send the corrected point cloud data to the server 150 for generation of the three-dimensional model 130 from the corrected point cloud data by the server 150. Alternatively, the server 150 may generate the three-dimensional model 130 directly from the corrected point cloud data 140 after correcting the point cloud data.

It is understood that the server 150 may be a background management server supporting the running of the client application in the electronic device 110, a virtual server, or the like, which is not limited in this disclosure.

It should be noted that the method for correcting the point cloud and/or the method for generating the three-dimensional model provided in the present disclosure may be executed by the electronic device 110, and may also be executed by the server 150. Accordingly, the correction device and/or the generation device of the three-dimensional model for providing the point cloud of the present disclosure may be disposed in the electronic device 110, and may also be disposed in the server 150.

It should be understood that the number and type of electronic devices 110 and servers 150 in fig. 1 are merely illustrative. There may be any number and type of electronic devices 110 and servers 150, as desired for an implementation.

The method for correcting the point cloud provided by the present disclosure will be described in detail below with reference to fig. 2 to 4.

Fig. 2 is a schematic flow chart diagram of a point cloud correction method according to an embodiment of the disclosure.

As shown in fig. 2, the method 200 for correcting the point cloud of this embodiment may include operations S210 to S250.

In operation S210, a plane fitting is performed according to the first point cloud data of the first object, so as to obtain a plane model representing a plane where the first object is located.

According to an embodiment of the present disclosure, the first point cloud data of the first object may be generated in advance from an image including the first object. For example, an image including a first object may be processed using a deep learning model to obtain point cloud data of the first object. The deep learning model can generate a network (3D-point-closed generation) for the 3D point cloud, the network predicts a 3D structure from multiple viewpoints using 2D convolution operation, and applies geometric inference and 2D projection optimization in a combined manner. The deep learning model may also be a Virtual 3D Scanner (Virtual 3D Scanner) that is used to simulate a real world 3D Scanner in a Virtual environment.

According to an embodiment of the present disclosure, a plane detection algorithm may be employed to perform plane fitting on the first point cloud data. The plane detection algorithm may include, for example, random Sample Consensus (RANSAC), or a hough transform-based plane detection algorithm.

Taking RANSAC as an example, the step of obtaining the plane model may include: firstly, randomly selecting three points from the space points represented By the first point cloud data, and calculating a plane equation Ax + By + Zz + D =0 corresponding to the three points. Then, algebraic distances of all spatial points of the first point cloud data representation to the plane are calculated, e.g., for the ith spatial point, from the planeDistance d _i ＝|Ax _i +By _i +Zz _i + D |. Then, a threshold value of the algebraic distance is chosen. If the algebraic distance between a certain space point and the plane is smaller than or equal to the threshold value, determining that the space point is an inner sample point of the plane model, and otherwise, determining that the space point is an outer sample point of the plane model. And repeating the steps, and selecting the best fitting parameters, namely the model parameters corresponding to the plane model with the largest number of the internal sample points. A plane model characterizing the plane in which the first object lies can be represented by the best fit parameters.

In operation S220, the first point cloud data is corrected according to the plane model to obtain corrected point cloud data.

The embodiment can determine a projection point of a space point represented by the first point cloud data on a plane represented by the plane model, and replace a coordinate value representing a space position of the space point in the first point cloud data with a coordinate value of the projection point, so as to realize the correction of the first point cloud data.

In operation S230, target point cloud data corresponding to a target boundary line in the target image in the corrected point cloud data is determined.

According to an embodiment of the present disclosure, the target boundary line is a boundary line between the first object and the second object in the target image. The target image includes a first object and a second object, and it can be understood that the first point cloud data may be point cloud data corresponding to a pixel point where the first object is located in the target image. Based on the target image, second point cloud data of a second object can also be obtained.

According to the embodiment of the disclosure, the target boundary line can be obtained by pre-labeling, and the target point cloud data can be selected from the point cloud data of the first object according to the corresponding relation between the pixel point of the local target boundary line and the first point cloud data.

According to embodiments of the present disclosure, the target boundary line may also be obtained by employing a HohoNet network framework. The Hohonet network framework is a multifunctional and efficient network framework which can comprehensively understand indoor 360-degree panorama by utilizing low-level ground full-view. The embodiment can input the target image into the network frame, and the network frame outputs the boundary line between the objects. For example, if the first object is a wall surface and the second object is a ground surface, the HohoNet network framework can be used to detect a boundary between the wall and the ground surface, which is a target boundary line. After the target boundary line is obtained, the target point cloud data corresponding to the target boundary line in the first point cloud data can be determined according to the corresponding relation between the pixels in the target image and the point cloud data.

In operation S240, based on a position relationship between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to the second object in the target image, a depth value corresponding to the second pixel point is determined according to the target point cloud data.

According to the embodiment of the disclosure, the first pixel point corresponding to the target point cloud data is the pixel point where the target boundary line described above is located.

According to an embodiment of the present disclosure, the center position of the lower edge line of the target image is set as a projection point of a center point of an image pickup device (e.g., a camera) on an image plane. In this embodiment, for each second pixel point, a pixel point located at the same height as that of each second pixel point in the image is determined from the first pixel points and used as a reference pixel point. The embodiment may then determine a pixel distance between the reference pixel point and each of the second pixel points. And deducing to obtain the depth value of each second pixel point by adopting a triangular measurement principle according to the pixel distance, the depth value of the space point corresponding to the reference pixel point and the projection point of the camera on the image plane. It can be understood that the method is based on the premise that a plurality of pixel points which are positioned at the same height in the height direction in the image are equal in vertical distance from the center point of the camera.

In an embodiment, the depth value corresponding to the second pixel point may also be determined by using the principle described below, for example, and the description is not expanded herein.

In operation S250, the second point cloud data of the second object is corrected according to the depth value.

It is to be understood that the second point cloud data of the second object may be generated from the target image in advance. The second point cloud data may be generated in advance, or may be generated in real time, which is not limited by the present disclosure. It is to be understood that the generation principle of the second point cloud data may be similar to that of the first point cloud data described above, and will not be described herein again.

In this embodiment, the depth value corresponding to the second pixel point obtained in operation S240 may be adopted to correct the depth value included in the second point cloud data corresponding to the second pixel point, and specifically, the depth value obtained in operation S240 may be adopted to replace the depth value included in the corresponding second point cloud data.

According to the method and the device, plane fitting is performed on the point cloud data of the first object, the point cloud data of the first object is adjusted according to a plane fitting result, the depth value of the pixel point corresponding to the second object is obtained through reasoning according to the point cloud data of the adjusted first object, the point cloud data of the second object is adjusted according to the depth value, the adjusted first point cloud data can accurately express the first object of a plane structure, and dislocation of the point cloud data between the first object and the second object expressed by the first point cloud data due to adjustment of the first point cloud data is compensated. Therefore, in the three-dimensional model generated based on the point cloud data after adjustment, no gap exists between the first object and the second object which are connected, and the objects are in a plane structure. Therefore, the precision of the corrected point cloud data expression scene can be improved, and the authenticity of the generated three-dimensional model is improved.

In an embodiment, before the point cloud data is corrected, for example, the point cloud data to be corrected may be generated according to the image, so as to facilitate implementation of the point cloud data correction process. The implementation principle of the correction method of the point cloud according to the embodiment of the present disclosure will be described below with reference to fig. 3.

Fig. 3 is an implementation schematic diagram of a point cloud correction method according to an embodiment of the present disclosure.

As shown in fig. 3, in this embodiment 300, before the point cloud data is corrected, a target image 310 including a first object and a second object may be acquired. The first object and the second object are two adjacent objects in the actual scene. The angle between the two objects may be, for example, approximately 90 °. For example, the first object may be a wall and the second object may be a floor; alternatively, the first object may be a floor and the second object may be a wall. It is to be understood that the first object and the second object are only examples to facilitate understanding of the present disclosure, and the first object and the second object may be any two objects having the same boundary line in an actual scene, which is not limited by the present disclosure.

As shown in fig. 3, after obtaining the target image 310, the embodiment may first generate the point cloud data 320 for the target image according to the target image 310. For example, the embodiment may first employ a sparse point cloud network to generate sparse point cloud data from each image, and then input the sparse point cloud data into a Dense model (Dense Module) to generate Dense point cloud. The sparse point cloud network may include, among other things, an encoder and a decoder. The encoder is composed of a convolution network, and the decoder is composed of a deconvolution network and a convolution network. The dense model may process the sparse point cloud data through a feature extraction operation and a feature expansion operation. The embodiment may take the dense point cloud data output by the dense model as the point cloud data 320 for the target image. It is to be appreciated that point cloud data corresponding to each pixel point in the target image 310 can be included in the point cloud data 320.

After the point cloud data 320 is obtained, the embodiment may determine the first point cloud data 321 in the point cloud data according to the pixel point (referred to as the third pixel point 311 for convenience of description) corresponding to the first object in the target image. In this embodiment, the target image 310 may be processed by using a target detection model, and the position of the first object in the target image 310 is detected, where the position may be represented by, for example, coordinate values of a bounding box of the first object. In this embodiment, the pixel points in the bounding box of the first object may be used as the pixel points corresponding to the first object. Then, the embodiment may determine the first point cloud data according to the correspondence between the pixel points and the point cloud data.

Similarly, after the point cloud data 320 is obtained, the embodiment may further determine the second point cloud data 322 in the point cloud data according to a pixel point (for convenience of description, referred to as a second pixel point 312) corresponding to the second object in the target image. It is understood that the determination principle of the second point cloud data 322 is similar to that of the first point cloud data 321, and is not described herein again.

After the first point cloud data 321 is obtained, a plane detection algorithm 330 may be used to perform plane fitting on the first point cloud data 321 to obtain a plane model 340. The obtaining principle of the plane model 340 is similar to the operation S210 described above, and is not described herein again.

After obtaining the plane model 340, the first point cloud data 321 may be corrected according to the plane model 340, so as to obtain corrected point cloud data 350. Subsequently, the embodiment may employ a principle similar to the above-described operations S230 to S240 to determine the depth value 360 corresponding to the second pixel point 312 according to the depth value included in the target point cloud data in the corrected point cloud data 350, and correct the second point cloud data 322 of the second object according to the depth value 360.

In an embodiment, when determining the first point cloud data 321 and the second point cloud data 322, the target object 310 may be processed by using a deep learning model, for example, to obtain a pixel position of a boundary line between the first object and an object other than the first object in the target image. It is understood that the boundary line between the first object and the other object includes the boundary line between the first object and the second object. The deep learning model may employ, for example, the HohoNet network framework described above. For example, if the first object is a wall, the detected boundary line may include a boundary line between the wall and a ceiling, a boundary line between the wall and a floor, and the like.

Subsequently, a pixel point corresponding to the first object in the target image can be determined according to the pixel position of the boundary line. For example, the pixel points included in the pixel region between the boundary line between the wall and the ground and the boundary line between the wall and the ceiling may be used as the pixel points corresponding to the first object. Meanwhile, in this embodiment, the pixel point corresponding to the second object in the target image 310 may also be determined according to the pixel position of the boundary line between the first object and the second object. For example, in the embodiment, a closed region surrounded by a boundary line between the first object and the second object and an edge line of the target image may be used as a region where the second object is located, and then the embodiment may use a pixel point in the closed region as a pixel point corresponding to the second object.

According to the embodiment, the boundary line is determined by adopting the deep learning model, the pixel points are determined according to the boundary line, and the accuracy of the obtained pixel points can be improved. Moreover, when the HohoNet network framework is adopted, the boundary line can be detected and obtained when the point cloud data aiming at the target image 310 is obtained, so that the implementation efficiency of the point cloud correction method can be improved, and the correction effect can be improved.

The principle of obtaining depth values in operation S240 described above will be further expanded and defined below.

Fig. 4 is a schematic diagram illustrating a principle of determining a depth value corresponding to a second pixel point according to an embodiment of the disclosure.

As shown in fig. 4, when determining the depth value corresponding to the second pixel point, the embodiment 400 may first determine a pixel point corresponding to the center of the virtual camera for the target image in the target image 410, and use the pixel point as a center pixel point. For example, if the target image 410 is a panoramic image, the center point 411 of the lower edge of the image may be used as a center pixel point, or the center point of the image may be used as a center pixel point, which is not limited in this disclosure. It can be understood that the pixel point corresponding to the virtual camera center of the target image may be understood as a projection point of the optical center of the camera that collects the target image on the phase plane of the target image. For example, in the target image 410, the curve 420 is a boundary line between the first object and the second object, and each pixel point where the curve 420 is located is the first pixel point described above.

After determining the central pixel point 411, the embodiment may determine a connection line between the central pixel point and each pixel point in the first pixel point, for example, to obtain a target connection line. For example, for the pixel 421 on the curve 420, the line between the pixel 421 and the central pixel 411 is a straight line 430.

After determining the straight line 430, the embodiment may determine a target pixel point located on the target connection line in the second pixel points corresponding to the second object. For example, the second pixel point located on the connection between the pixel point 421 and the center pixel point 411 may include a plurality of pixel points such as the pixel point 441.

The embodiment 400 may determine the depth value of the target pixel according to the pixel distance between each pixel in the first pixel and the target pixel on the target link where the first pixel is located. For example, the depth value of the target pixel point may be determined according to the depth value included in the target point cloud data of the pixel point.

For example, for the pixel 441, the depth value of the pixel 441 can be determined according to a pixel distance between the pixel 441 and the pixel 421 and a depth value included in the point cloud data corresponding to the pixel 421. For example, the distance between the pixel 441 and the pixel 421 is set to D, and the depth value D included in the point cloud data corresponding to the pixel 421 is set ₁ The depth value corresponding to the central pixel point is a predetermined value d ₀ Then the depth value d corresponding to the pixel 441 is obtained ₂ The following formula can be used to calculate:

d ₂ ＝d ₁ +d ₁ *D/(d ₁ -d ₀ )。

specifically, in one embodiment, the depth value corresponding to the central pixel point is set to a predetermined value d ₀ Then, the killing power may determine the corresponding depth value of the target pixel point by using a linear interpolation algorithm according to the pixel distance and the depth value included in the target point cloud data corresponding to each pixel point.

It is to be understood that the above-mentioned principle of determining the depth value corresponding to the target pixel point is only used as an example to facilitate understanding of the present disclosure, and the present disclosure is not limited thereto.

Based on the point cloud correction method provided by the present disclosure, the present disclosure also provides a three-dimensional model generation method, which will be described in detail below with reference to fig. 5 to 6.

Fig. 5 is a flow chart diagram of a method of generating a three-dimensional model according to an embodiment of the disclosure.

As shown in fig. 5, the generation method 500 of the three-dimensional model of this embodiment may include operations S510 to S530.

In operation S510, initial point cloud data corresponding to a target image is generated according to the target image including a first object and a second object.

It will be appreciated that the initial point cloud data includes first point cloud data for a first object and second point cloud data for a second object. The principle of generating the initial point cloud data in this embodiment may be similar to the principle of generating the point cloud data for the target image described above, for example, and is not described herein again.

After obtaining the initial point cloud data, the embodiment may also determine the first point cloud data of the first object and the second point cloud data of the second object in the initial point cloud data by using a similar method as described above.

In operation S520, the first point cloud data and the second point cloud data are corrected to obtain corrected point cloud data.

In operation S530, a three-dimensional model is generated according to the corrected point cloud data.

In operation S520, the point cloud data of the first point and the point cloud data of the second point may be corrected by using the point cloud correction method described above, which is not described herein again. The embodiment may represent a three-dimensional model of a scene described by the target image from the corrected point cloud data.

According to the method for generating the three-dimensional model, after the point cloud data are obtained, the point cloud data are corrected, and then the corrected point cloud data are adopted to represent the three-dimensional model, so that the authenticity of the obtained three-dimensional model can be improved. This is because the first point cloud data of the first object is corrected based on the plane model obtained by plane fitting, so that it can be ensured that the first object in the obtained three-dimensional model is of a planar structure, and the three-dimensional model can more truly express the three-dimensional structure of the first object.

The effect of the three-dimensional model generated from the corrected point cloud data will be described below with reference to fig. 6.

Fig. 6 is an effect diagram of a three-dimensional model generated by a method of generating a three-dimensional model according to an embodiment of the present disclosure.

As shown in fig. 6, in embodiment 600, a first object is set as a wall, a second object is set as a ground, and if a three-dimensional model is generated according to the initial point cloud data, the obtained three-dimensional model is a model 610. If the point cloud correction method provided by the present disclosure is adopted to correct the first point cloud data and the second point cloud data, and then the three-dimensional model is generated according to the corrected point cloud data, the model 620 may be obtained.

It can be seen that the wall 611 in the model 610 and the boundary between the wall 611 and the ground 612 are distorted and do not conform to the real scene. By correcting the point cloud data of the wall surface, the wall surface 621 in the generated model 620 is a planar structure and conforms to a real scene. Moreover, by correcting the point cloud data of the ground based on the point cloud data of the corrected wall surface, the seam between the ground 622 and the wall surface 621 can be made complete, and the situation of having a gap does not exist, and the corrected point cloud data of the ground can be attached to a real scene.

Based on the point cloud correction method provided by the disclosure, the disclosure also provides a point cloud correction device. The apparatus will be described in detail below with reference to fig. 7.

Fig. 7 is a block diagram of a structure of a point cloud correction apparatus according to an embodiment of the present disclosure.

As shown in fig. 7, the apparatus 700 for correcting a point cloud of this embodiment may include a plane fitting module 710, a first correction module 720, a boundary point cloud determination module 730, a depth value determination module 740, and a second correction module 750.

The plane fitting module 710 is configured to perform plane fitting according to the first point cloud data of the first object to obtain a plane model representing a plane where the first object is located. In an embodiment, the plane fitting module 710 may be configured to perform the operation S210 described above, which is not described herein again.

The first correction module 720 is configured to correct the first point cloud data according to the plane model, so as to obtain corrected point cloud data. In an embodiment, the first calibration module 720 may be configured to perform the operation S220 described above, which is not described herein again.

The boundary point cloud determining module 730 is configured to determine target point cloud data corresponding to a target boundary line in the target image in the corrected point cloud data, where the target boundary line is a boundary line between a first object and a second object in the target image. In an embodiment, the boundary point cloud determining module 730 may be configured to perform the operation S230 described above, which is not described herein again.

The depth value determining module 740 is configured to determine a depth value corresponding to a second pixel point according to the target point cloud data based on a position relationship between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to a second object in the target image. In an embodiment, the depth value determining module 740 may be configured to perform the operation S240 described above, which is not described herein again.

The second correction module 750 is configured to correct the second point cloud data of the second object according to the depth value. In an embodiment, the second calibration module 750 can be used to perform the operation S250 described above, which is not described herein again.

According to an embodiment of the present disclosure, the depth value determining module 740 may include: the central point determining submodule is used for determining a pixel point corresponding to the center of the virtual camera aiming at the target image in the target image as a central pixel point; the connecting line obtaining submodule is used for determining a connecting line between the central pixel point and each pixel point in the first pixel points to obtain a target connecting line; the target point determining submodule is used for determining a target pixel point which is positioned on a target connecting line in the second pixel points; and the depth determining submodule is used for determining the corresponding depth value of the target pixel point according to the target point cloud data corresponding to each pixel point based on the pixel distance between each pixel point and the target pixel point.

According to an embodiment of the disclosure, the depth determination submodule is to: and determining the corresponding depth value of the target pixel point by adopting a linear interpolation algorithm according to the pixel distance and the depth value in the target point cloud data corresponding to each pixel point. And the depth value corresponding to the central pixel point is a preset value.

According to an embodiment of the present disclosure, the apparatus 700 may further include: the point cloud generating module is used for generating point cloud data aiming at the target image according to the target image; the first point cloud determining module is used for determining first point cloud data in the point cloud data according to pixel points corresponding to the first object in the target image; and the second point cloud determining module is used for determining second point cloud data in the point cloud data according to the pixel points corresponding to the second object in the target image.

According to an embodiment of the present disclosure, the apparatus 700 may further include: the boundary line determining module is used for processing the target image by adopting a deep learning model to obtain the pixel position of the boundary line between the first object and other objects except the first object in the target image; the boundary lines between the first object and the other objects include boundary lines between the first object and the second object; the first pixel point determining module is used for determining pixel points corresponding to a first object in the target image according to the pixel positions; and the second pixel point determining module is used for determining the pixel point corresponding to the second object in the target image according to the pixel position of the boundary line between the first object and the second object.

According to an embodiment of the present disclosure, one of the first object and the second object is a wall; the other of the first object and the second object is the ground.

Based on the three-dimensional model generation method provided by the disclosure, the disclosure also provides a three-dimensional model generation device. The apparatus will be described in detail below with reference to fig. 8.

Fig. 8 is a block diagram of a three-dimensional model generation apparatus according to an embodiment of the present disclosure.

As shown in fig. 8, the generation apparatus 800 of the three-dimensional model of this embodiment may include a point cloud generation module 810, a point cloud correction module 820, and a model generation module 830.

The point cloud generating module 810 is configured to generate initial point cloud data corresponding to a target image according to the target image including a first object and a second object; the initial point cloud data includes first point cloud data of a first object and second point cloud data of a second object. In an embodiment, the point cloud generating module 810 may be configured to perform the operation S510 described above, which is not described herein again.

The point cloud correction module 820 is configured to correct the first point cloud data and the second point cloud data by using the point cloud correction device provided by the present disclosure, so as to obtain corrected point cloud data. In an embodiment, the point cloud correction module 820 may be configured to perform the operation S520, which is not described herein again.

The model generation module 830 is configured to generate a three-dimensional model according to the corrected point cloud data. In an embodiment, the model generating module 830 may be configured to perform the operation S530 described above, which is not described herein again.

In the technical scheme of the present disclosure, the processes of collecting, storing, using, processing, transmitting, providing, disclosing and applying the personal information of the user all conform to the regulations of the relevant laws and regulations, and necessary security measures are taken without violating the customs of the public order. In the technical scheme of the disclosure, before the personal information of the user is acquired or collected, the authorization or the consent of the user is acquired.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

Fig. 9 illustrates a schematic block diagram of an example electronic device 900 that may be used to implement the correction methods of point clouds and/or the generation methods of three-dimensional models of embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 9, the apparatus 900 includes a computing unit 901, which can perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM) 902 or a computer program loaded from a storage unit 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The calculation unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.

A number of components in the device 900 are connected to the I/O interface 905, including: an input unit 906 such as a keyboard, a mouse, and the like; an output unit 907 such as various types of displays, speakers, and the like; a storage unit 908 such as a magnetic disk, optical disk, or the like; and a communication unit 909 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the device 900 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The computing unit 901 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 901 performs the respective methods and processes described above, such as the correction method of the point cloud and/or the generation method of the three-dimensional model. For example, in some embodiments, the correction method of the point cloud and/or the generation method of the three-dimensional model may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 900 via ROM 902 and/or communications unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the above-described correction method of the point cloud and/or generation method of the three-dimensional model may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured by any other suitable means (e.g. by means of firmware) to perform a correction method of the point cloud and/or a generation method of the three-dimensional model.

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service extensibility in a traditional physical host and a VPS service ("Virtual Private Server", or simply "VPS"). The server may also be a server of a distributed system, or a server incorporating a blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method of correcting a point cloud, comprising:

performing plane fitting according to the first point cloud data of the first object to obtain a plane model representing the plane where the first object is located;

correcting the first point cloud data according to the plane model to obtain corrected point cloud data;

determining target point cloud data corresponding to a target boundary line in a target image in the corrected point cloud data, wherein the target boundary line is a boundary line between the first object and the second object in the target image;

determining a depth value corresponding to a second pixel point according to the target point cloud data based on a position relation between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to the second object in the target image; and

and correcting the second point cloud data of the second object according to the depth value.

2. The method of claim 1, wherein the determining a depth value corresponding to the second pixel point from the target point cloud data based on a positional relationship between a first pixel point corresponding to the target point cloud data in the target image and a second pixel point corresponding to the second object in the target image comprises:

determining a pixel point in the target image corresponding to the virtual camera center for the target image as a center pixel point;

determining a connecting line between the central pixel point and each pixel point in the first pixel points to obtain a target connecting line;

determining a target pixel point positioned on the target connecting line in the second pixel points; and

and determining the corresponding depth value of the target pixel point according to the target point cloud data corresponding to each pixel point based on the pixel distance between each pixel point and the target pixel point.

3. The method of claim 2, wherein the determining, based on the pixel distance between the each pixel point and the target pixel point, a depth value corresponding to the target pixel point from the target point cloud data corresponding to the each pixel point comprises:

determining the corresponding depth value of the target pixel point by adopting a linear interpolation algorithm according to the pixel distance and the depth value in the target point cloud data corresponding to each pixel point,

and the depth value corresponding to the central pixel point is a preset value.

4. The method of claim 1, further comprising:

generating point cloud data for the target image according to the target image;

determining the first point cloud data in the point cloud data according to the pixel point corresponding to the first object in the target image; and

and determining the second point cloud data in the point cloud data according to the pixel point corresponding to the second object in the target image.

5. The method of claim 4, further comprising:

processing the target image by adopting a deep learning model to obtain the pixel position of a boundary line between the first object and other objects except the first object in the target image; a boundary line between the first object and the other object includes a boundary line between the first object and the second object;

determining pixel points corresponding to the first object in the target image according to the pixel positions; and

and determining pixel points corresponding to the second object in the target image according to the pixel positions of the boundary line between the first object and the second object.

6. The method of any of claims 1-5, wherein one of the first object and the second object is a wall; the other of the first object and the second object is the ground.

7. A method of generating a three-dimensional model, comprising:

generating initial point cloud data corresponding to a target image according to the target image comprising a first object and a second object; the initial point cloud data comprises first point cloud data of the first object and second point cloud data of the second object;

correcting the first point cloud data and the second point cloud data by using the point cloud correction method of any one of claims 1 to 6 to obtain corrected point cloud data; and

and generating the three-dimensional model according to the corrected point cloud data.

8. A device for correcting a point cloud, comprising:

the plane fitting module is used for performing plane fitting according to the first point cloud data of the first object to obtain a plane model representing the plane of the first object;

the first correction module is used for correcting the first point cloud data according to the plane model to obtain corrected point cloud data;

a boundary point cloud determining module, configured to determine target point cloud data corresponding to a target boundary line in a target image in the corrected point cloud data, where the target boundary line is a boundary line between the first object and a second object in the target image;

a depth value determining module, configured to determine, according to the target point cloud data, a depth value corresponding to a second pixel point corresponding to the second object in the target image based on a position relationship between the first pixel point corresponding to the target point cloud data in the target image and the second pixel point corresponding to the second object in the target image; and

and the second correction module is used for correcting the second point cloud data of the second object according to the depth value.

9. The apparatus of claim 8, wherein the depth value determination module comprises:

the central point determining submodule is used for determining a pixel point corresponding to the virtual camera center aiming at the target image in the target image as a central pixel point;

the connecting line obtaining submodule is used for determining a connecting line between the central pixel point and each pixel point in the first pixel points to obtain a target connecting line;

the target point determining submodule is used for determining a target pixel point which is positioned on the target connecting line in the second pixel points; and

and the depth determining submodule is used for determining the corresponding depth value of the target pixel point according to the target point cloud data corresponding to each pixel point based on the pixel distance between each pixel point and the target pixel point.

10. The apparatus of claim 9, wherein the depth determination submodule is to:

determining the depth value corresponding to the target pixel point by adopting a linear interpolation algorithm according to the pixel distance and the depth value in the target point cloud data corresponding to each pixel point,

and the depth value corresponding to the central pixel point is a preset value.

11. The apparatus of claim 8, further comprising:

the point cloud generating module is used for generating point cloud data aiming at the target image according to the target image;

the first point cloud determining module is used for determining the first point cloud data in the point cloud data according to the pixel point corresponding to the first object in the target image; and

and the second point cloud determining module is used for determining the second point cloud data in the point cloud data according to the pixel points corresponding to the second object in the target image.

12. The apparatus of claim 11, further comprising:

the boundary line determining module is used for processing the target image by adopting a deep learning model to obtain the pixel position of the boundary line between the first object and other objects except the first object in the target image; boundary lines between the first object and the other objects include boundary lines between the first object and the second object;

the first pixel point determining module is used for determining pixel points corresponding to the first object in the target image according to the pixel positions; and

and the second pixel point determining module is used for determining a pixel point corresponding to the second object in the target image according to the pixel position of the boundary line between the first object and the second object.

13. The apparatus of any one of claims 8-12, wherein one of the first object and the second object is a wall; the other of the first object and the second object is the ground.

14. An apparatus for generating a three-dimensional model, comprising:

the point cloud generating module is used for generating initial point cloud data corresponding to a target image according to the target image comprising a first object and a second object; the initial point cloud data comprises first point cloud data of the first object and second point cloud data of the second object;

a point cloud correction module, configured to correct the first point cloud data and the second point cloud data by using the apparatus according to any one of claims 8 to 13, so as to obtain corrected point cloud data; and

and the model generation module is used for generating the three-dimensional model according to the corrected point cloud data.

15. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method according to any one of claims 1 to 7.

17. A computer program product comprising computer program/instructions stored on at least one of a readable storage medium and an electronic device, which when executed by a processor implement the steps of the method according to any one of claims 1 to 7.