CN112154485A

CN112154485A - Optimization method and equipment of three-dimensional reconstruction model and movable platform

Info

Publication number: CN112154485A
Application number: CN201980033605.1A
Authority: CN
Inventors: 胡鑫
Original assignee: SZ DJI Technology Co Ltd
Current assignee: SZ DJI Technology Co Ltd
Priority date: 2019-08-30
Filing date: 2019-08-30
Publication date: 2020-12-29
Also published as: WO2021035749A1

Abstract

A method, equipment and a movable platform for optimizing a three-dimensional reconstruction model are disclosed, wherein the method comprises the following steps: acquiring a reprojection error of each patch in a three-dimensional reconstruction model to be optimized of a target object projected to M reference images (S201); determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively (S202); if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain an optimized three-dimensional reconstruction model (S203). The three-dimensional reconstruction model is a model that is generated from N initial images including at least a partial image of a target object and that is composed of a plurality of patches, and the reference image is an image obtained by downsampling L levels of the initial images. The optimization efficiency of the three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced.

Description

Optimization method and equipment of three-dimensional reconstruction model and movable platform

Technical Field

The embodiment of the application relates to the technical field of images, in particular to an optimization method and equipment of a three-dimensional reconstruction model and a movable platform.

Background

The three-dimensional reconstruction is an important research direction in the field of image vision, and refers to acquiring a plurality of two-dimensional images of a scene three-dimensional object through a camera device, analyzing and processing the images, and deducing three-dimensional information of the object in a real environment by combining computer vision knowledge, wherein the three-dimensional information can be used for generating a three-dimensional reconstruction model of the object. The three-dimensional reconstruction model has the characteristics incomparable with two-dimensional images, and can be visually observed from a plurality of different angles so as to achieve the effects of vividness, real-time virtualization, real-time interaction and the like.

The three-dimensional reconstruction model is a triangular mesh curved surface with texture, and a user can perform operations such as measurement, rendering, 3D printing and the like by using the triangular mesh after obtaining the three-dimensional reconstruction model, so that the triangular mesh is one of important indexes for measuring the three-dimensional reconstruction model. Therefore, the triangular mesh curved surface needs to be optimized after the three-dimensional reconstruction model is obtained.

Disclosure of Invention

The embodiment of the application provides an optimization method and equipment of a three-dimensional reconstruction model and a movable platform, which are used for improving the optimization efficiency of the three-dimensional reconstruction model.

In a first aspect, an embodiment of the present application provides an optimization method for a three-dimensional reconstruction model, including:

acquiring a reprojection error of each patch in a three-dimensional reconstruction model to be optimized of a target object projected to M reference images respectively;

determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively;

if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain an optimized three-dimensional reconstruction model;

the three-dimensional reconstruction model is generated according to N initial images and is composed of a plurality of patches, the N initial images at least comprise partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 2 and less than or equal to N.

In a second aspect, an embodiment of the present application provides an optimization method for a three-dimensional reconstruction model, including:

acquiring texture complexity of each patch in a three-dimensional reconstruction model to be optimized of a target object in a corresponding area on M reference images;

determining whether the patch is a patch to be optimized according to the texture complexity of the patch in the corresponding area on the M reference images;

the three-dimensional reconstruction model is generated according to N initial images and is composed of a plurality of patches, the N initial images at least comprise partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 2, and M is an integer greater than or equal to 2 and less than or equal to N.

In a third aspect, an embodiment of the present application provides an optimization apparatus for a three-dimensional reconstruction model, including:

the shooting device is used for acquiring an initial image;

the processor is used for acquiring a reprojection error of each patch in the three-dimensional reconstruction model to be optimized of the target object projected to M reference images respectively; determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively; if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain an optimized three-dimensional reconstruction model;

the three-dimensional reconstruction model is generated by the processor according to N initial images shot by the shooting device and is composed of a plurality of patches, the N initial images at least comprise partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 2 and less than or equal to N.

In a fourth aspect, an embodiment of the present application provides an optimization apparatus for a three-dimensional reconstruction model, including:

the shooting device is used for acquiring an initial image;

the processor is used for acquiring the texture complexity of each patch in the three-dimensional reconstruction model to be optimized of the target object in a corresponding area on the M reference images; determining whether the patch is a patch to be optimized according to the texture complexity of the patch in the corresponding area on the M reference images; if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain an optimized three-dimensional reconstruction model;

the three-dimensional reconstruction model is generated by the processor according to N initial images acquired by the shooting device and is composed of a plurality of patches, the N initial images at least comprise partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 2, and M is an integer greater than or equal to 2 and less than or equal to N.

In a fifth aspect, an embodiment of the present application provides a movable platform, including: the three-dimensional reconstruction model optimization device comprises a movable platform body and the three-dimensional reconstruction model optimization device according to the first aspect, wherein the three-dimensional reconstruction model optimization device is installed on the movable platform body.

In a sixth aspect, an embodiment of the present application provides a movable platform, including: the three-dimensional reconstruction model optimization device comprises a movable platform body and the three-dimensional reconstruction model optimization device according to the second aspect, wherein the three-dimensional reconstruction model optimization device is installed on the movable platform body.

In a seventh aspect, an embodiment of the present application provides a readable storage medium, where a computer program is stored on the readable storage medium; the computer program, when executed, implements a method for optimizing a three-dimensional reconstruction model according to an embodiment of the present application in the first or second aspect.

In an eighth aspect, the present application provides a program product, where the program product includes a computer program stored in a readable storage medium, and the computer program can be read by at least one processor of a movable platform from the readable storage medium, and the computer program is executed by the at least one processor to enable the movable platform to implement the method for optimizing a three-dimensional reconstruction model according to the first aspect or the second aspect.

According to the optimization method, the optimization device and the movable platform of the three-dimensional reconstruction model, all the patches are not directly optimized, but the determined patches to be optimized are optimized, so that the optimization efficiency of the three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic architecture diagram of an unmanned flight system according to an embodiment of the present application;

fig. 2 is a flowchart of an optimization method for a three-dimensional reconstruction model according to an embodiment of the present disclosure;

fig. 3 is a flowchart of a method for optimizing a three-dimensional reconstruction model according to another embodiment of the present application;

fig. 4 is a schematic diagram of a method for dividing a patch into H patches according to an embodiment of the present application;

fig. 5 is a schematic diagram illustrating relationships between vertices and patches to which the vertices belong according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of an optimization apparatus for a three-dimensional reconstruction model according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an optimization apparatus for a three-dimensional reconstruction model according to another embodiment of the present application;

FIG. 8 is a schematic structural diagram of a movable platform according to an embodiment of the present disclosure;

FIG. 9 is a schematic structural diagram of a movable platform according to another embodiment of the present application;

fig. 10 is a schematic structural diagram of a movable platform according to another embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The embodiment of the application provides an optimization method and equipment of a three-dimensional reconstruction model and a movable platform, wherein the movable platform can be a handheld phone, a handheld cloud deck, an unmanned aerial vehicle, an unmanned ship, a robot or an automatic driving automobile and the like. The following description of the movable platform of the present application uses a drone as an example. It will be apparent to those skilled in the art that other types of drones may be used without limitation, and embodiments of the present application may be applied to various types of drones. For example, the drone may be a small or large drone. In certain embodiments, the drone may be a rotorcraft (rotorcraft), for example, a multi-rotor drone propelled through the air by a plurality of propulsion devices, embodiments of the present application are not so limited, and the drone may be other types of drones as well.

Fig. 1 is a schematic architecture diagram of an unmanned flight system according to an embodiment of the present application. The present embodiment is described by taking a rotor unmanned aerial vehicle as an example.

The unmanned flight system 100 can include a drone 110, a display device 130, and a remote control device 140. The drone 110 may include, among other things, a power system 150, a flight control system 160, a frame, and a pan-tilt 120 carried on the frame. The drone 110 may be in wireless communication with the remote control device 140 and the display device 130.

The airframe may include a fuselage and a foot rest (also referred to as a landing gear). The fuselage may include a central frame and one or more arms connected to the central frame, the one or more arms extending radially from the central frame. The foot rest is connected with the fuselage for play the supporting role when unmanned aerial vehicle 110 lands.

The power system 150 may include one or more electronic governors (abbreviated as electric governors) 151, one or more propellers 153, and one or more motors 152 corresponding to the one or more propellers 153, wherein the motors 152 are connected between the electronic governors 151 and the propellers 153, the motors 152 and the propellers 153 are disposed on the horn of the drone 110; the electronic governor 151 is configured to receive a drive signal generated by the flight control system 160 and provide a drive current to the motor 152 based on the drive signal to control the rotational speed of the motor 152. The motor 152 is used to drive the propeller in rotation, thereby providing power for the flight of the drone 110, which power enables the drone 110 to achieve one or more degrees of freedom of motion. In certain embodiments, the drone 110 may rotate about one or more axes of rotation. For example, the above-mentioned rotation axes may include a Roll axis (Roll), a Yaw axis (Yaw) and a pitch axis (pitch). It should be understood that the motor 152 may be a dc motor or an ac motor. The motor 152 may be a brushless motor or a brush motor.

Flight control system 160 may include a flight controller 161 and a sensing system 162. The sensing system 162 is used to measure attitude information of the drone, i.e., position information and status information of the drone 110 in space, such as three-dimensional position, three-dimensional angle, three-dimensional velocity, three-dimensional acceleration, three-dimensional angular velocity, and the like. The sensing system 162 may include, for example, at least one of a gyroscope, an ultrasonic sensor, an electronic compass, an Inertial Measurement Unit (IMU), a vision sensor, a global navigation satellite system, and a barometer. For example, the Global navigation satellite System may be a Global Positioning System (GPS). The flight controller 161 is used to control the flight of the drone 110, for example, the flight of the drone 110 may be controlled according to attitude information measured by the sensing system 162. It should be understood that the drone 110 may be controlled by the flight controller 161 in accordance with preprogrammed instructions, or the drone 110 may be controlled in response to one or more remote control signals from the remote control device 140.

The pan/tilt head 120 may include a motor 122. The pan/tilt head is used to carry the photographing device 123. Flight controller 161 may control the movement of pan/tilt head 120 via motor 122. Optionally, as another embodiment, the pan/tilt head 120 may further include a controller for controlling the movement of the pan/tilt head 120 by controlling the motor 122. It should be understood that the pan/tilt head 120 may be separate from the drone 110, or may be part of the drone 110. It should be understood that the motor 122 may be a dc motor or an ac motor. The motor 122 may be a brushless motor or a brush motor. It should also be understood that the pan/tilt head may be located at the top of the drone, as well as at the bottom of the drone.

The photographing device 123 may be, for example, a device for capturing an image such as a camera or a video camera, and the photographing device 123 may communicate with the flight controller and perform photographing under the control of the flight controller. The image capturing Device 123 of this embodiment at least includes a photosensitive element, such as a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge-coupled Device (CCD) sensor. It can be understood that the camera 123 may also be directly fixed to the drone 110, such that the pan/tilt head 120 may be omitted.

The display device 130 is located at the ground end of the unmanned aerial vehicle system 100, can communicate with the unmanned aerial vehicle 110 in a wireless manner, and can be used for displaying attitude information of the unmanned aerial vehicle 110. In addition, an image photographed by the photographing device may also be displayed on the display apparatus 130. It should be understood that the display device 130 may be a stand-alone device or may be integrated into the remote control device 140.

The remote control device 140 is located at the ground end of the unmanned aerial vehicle system 100, and can communicate with the unmanned aerial vehicle 110 in a wireless manner, so as to remotely control the unmanned aerial vehicle 110.

It should be understood that the above-mentioned nomenclature for the components of the unmanned flight system is for identification purposes only, and should not be construed as limiting the embodiments of the present application.

Fig. 2 is a flowchart of an optimization method for a three-dimensional reconstruction model according to an embodiment of the present application, and as shown in fig. 2, the method according to the embodiment may include:

s201, obtaining a re-projection error of each patch in the three-dimensional reconstruction model to be optimized of the target object projected to M reference images respectively.

S202, determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively.

S203, if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

In this embodiment, the three-dimensional reconstruction model to be optimized of the target object is a model that is generated from N initial images and is composed of a plurality of patches, where the N initial images at least include a partial image of the target object, and N is an integer greater than or equal to 3. Optionally, the patch is, for example, a triangular patch. Geometric details on the three-dimensional reconstruction model may be unevenly distributed, some patches have few geometric details, and some patches have rich geometric details, so that the patches with few geometric details can be optimized, and the geometric details are distributed as evenly as possible.

Firstly, obtaining a reprojection error of each patch in a three-dimensional reconstruction model to be optimized of a target object projected to M reference images respectively, wherein the reference images are images obtained by downsampling L levels of the initial image, L is an integer larger than or equal to 0, and M is an integer larger than or equal to 2 and smaller than or equal to N. Wherein, when N is equal to 3, M is equal to 2. The M reference images are M images obtained by respectively sampling L levels of M initial images in the N initial images.

Optionally, before performing S201 described above, L levels are also downsampled for each of M initial images of the N initial images to obtain M reference images.

Optionally, before performing S201 above, each of the N initial images is also downsampled by L levels to obtain N reference images, and then the areas of the pixel regions of the patches respectively projected onto the N reference images are obtained. If the area of the pixel region projected by the patch into the reference image is larger than a preset value, acquiring a re-projection error projected by the patch into the reference image, and if the area of the pixel region projected by the patch into the reference image is not larger than the preset value, not acquiring the re-projection error projected by the patch into the reference image. And if the areas of the pixel regions respectively projected to the N reference images by the patch are all larger than a preset value, the M is equal to N. And if the areas of the pixel regions respectively projected to the N reference images by the patch are not all larger than a preset value, the M is smaller than the N. Wherein, the preset value is determined according to the area of the patch.

In the following, one patch is taken as an example, and the processing of the other patches is the same. And after the reprojection errors of the patches projected into the M reference images are obtained, determining whether the patches are to-be-optimized patches according to the reprojection errors of the patches projected into the M reference images. And if the patch is determined to be the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model, wherein the patch after the position adjustment is the optimized patch so as to obtain the optimized three-dimensional reconstruction model. If the patch is determined not to be the patch to be optimized, the position of the patch in the three-dimensional reconstruction model is not required to be adjusted, namely the position of the patch in the three-dimensional reconstruction model is kept unchanged.

Optionally, after determining that the patch is the patch to be optimized, the patch may be identified, that is, the patch is identified as the patch to be optimized, for example, the patch is identified as active, and accordingly, the position of the patch identified as active in the three-dimensional reconstruction model is adjusted.

Optionally, after determining that the patch is not the patch to be optimized, the patch may be identified, that is, the patch is identified as not the patch to be optimized, for example, the patch is identified as inactive, and accordingly, the position of the patch identified as inactive in the three-dimensional reconstruction model is kept unchanged.

In the optimization method of the three-dimensional reconstruction model provided by this embodiment, a reprojection error in which each patch in the three-dimensional reconstruction model to be optimized of the target object is projected to M reference images is obtained; determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively; and if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model. Because the embodiment does not directly optimize all patches, but optimizes the determined patches to be optimized, the optimization efficiency of the three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced.

In some embodiments, one possible implementation manner of the above S202 may include S2021 and S2022:

s2021, determining gradients of the reprojected areas of the patches on the M reference images according to the reprojection errors of the patches projected to the M reference images respectively.

S2022, determining whether the patch is a patch to be optimized according to the gradient of the reprojected areas of the patch on the M reference images.

In this embodiment, according to the reprojection errors of the patch projected onto the M reference images, the reprojected regions of the patch on the M reference images can be determined, and then the gradients of the reprojected regions of the patch on the M reference images can be determined. One possible implementation of determining the gradient of the reprojected region is: the gradient of each point in the reprojected region is determined, and then the gradient of the patch in the reprojected region is determined based on the gradients of all points in the reprojected region. For example, the gradient of the patch in the reprojected region may be the sum of the gradients of all points in the reprojected region.

Then, according to the gradient of the area of the re-projection of the patch on M reference images, it is determined whether the patch is a patch to be optimized, for example: determining the target gradient of the patch according to the gradients of the reprojected areas of the patch on the M reference images respectively; and then determining whether the patch is a patch to be optimized according to the comparison result of the target gradient and the preset gradient, for example: and judging whether the target gradient of the surface patch is greater than a preset gradient, if so, determining that the surface patch is the surface patch to be optimized, and if not, determining that the surface patch is not the surface patch to be optimized.

Optionally, the target gradient comprises one of: the maximum gradient of the gradients of the reprojected regions of the patch on the M reference images, the minimum gradient of the gradients of the reprojected regions of the patch on the M reference images, and the average gradient of the gradients of the reprojected regions of the patch on the M reference images. Note that the target gradient of the present embodiment is not limited to this.

According to the gradient of the reprojected areas of the patches on the M reference images, the patch to be optimized can be determined more accurately, the optimization efficiency of the three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced.

In some embodiments, one possible implementation of S202 described above may include S2021 'and S2022':

s2021', determining the target reprojection error of the patch according to the reprojection errors of the patch projected to M reference images respectively.

S2022', according to the target re-projection error, determining whether the patch is a patch to be optimized.

In this embodiment, a target reprojection error of a patch can be determined according to the reprojection errors of the patch projected into M reference images, where the target reprojection error includes, for example, one of the following: the maximum reprojection error of the patch in the M reference images, the minimum reprojection error of the patch in the M reference images, and the average reprojection error of the patch in the M reference images. It should be noted that the target reprojection error of the present embodiment is not limited thereto. Then, according to the target re-projection error, determining whether the patch is a patch to be optimized, wherein one implementation manner may be: determining whether the patch is a patch to be optimized according to the comparison result of the target reprojection error and the preset reprojection error, for example: and judging whether the target re-projection error of the patch is larger than a preset re-projection error or not, if so, determining that the patch is the patch to be optimized, and if not, determining that the patch is not the patch to be optimized.

Alternatively, the reprojection error of the patch projected onto each of the M reference images, respectively, may be represented by the sum of the gradients of the points in the pixel region of the patch projected onto the reference image. That is, the reprojection error corresponding to a patch is correlated, e.g., positively correlated, with the gradient of the reprojected region corresponding to the patch.

In this embodiment, a target reprojection error of the patch is determined according to reprojection errors of the patch projected into M reference images, and whether the patch is a patch to be optimized is determined according to the target reprojection error, so that the patch to be optimized can be determined more accurately, the optimization efficiency of the three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced. For example: if the target reprojection error is the maximum reprojection error of the patch in the M reference images, the larger the maximum reprojection error is, the larger the patch error is, the more the patch error should be optimized.

Fig. 3 is a flowchart of a method for optimizing a three-dimensional reconstruction model according to another embodiment of the present disclosure, and as shown in fig. 3, the method according to this embodiment may include:

s301, obtaining texture complexity of each patch in the three-dimensional reconstruction model to be optimized of the target object in a corresponding area on the M reference images.

S302, determining whether the patch is a patch to be optimized according to the texture complexity of the corresponding areas of the patch on the M reference images.

And S303, if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

In this embodiment, the three-dimensional reconstruction model to be optimized of the target object is a model that is generated from N initial images and is composed of a plurality of patches, where the N initial images at least include a partial image of the target object, and N is an integer greater than or equal to 2. Optionally, the patch is, for example, a triangular patch. Geometric details on the three-dimensional reconstruction model may be unevenly distributed, some patches have few geometric details, and some patches have rich geometric details, so that patches with more geometric details can be optimized, and the geometric details are distributed as evenly as possible.

The method comprises the steps of firstly, obtaining texture complexity of corresponding areas of each patch in a three-dimensional reconstruction model to be optimized of a target object on M reference images, wherein the reference images are obtained by sampling L levels of the initial image, L is an integer larger than or equal to 0, and M is an integer larger than or equal to 2 and smaller than or equal to N. The M reference images are M images obtained by respectively sampling L levels of M initial images in the N initial images.

Optionally, before performing S301 above, L levels are also downsampled for each of M initial images of the N initial images to obtain M reference images.

Optionally, before performing S301 above, each of the N initial images is also downsampled by L levels to obtain N reference images, and then the areas of the pixel regions of the patches respectively projected onto the N reference images are obtained. If the area of the pixel region projected by the patch into the reference image is larger than a preset value, the texture complexity of the corresponding region of the patch on the reference image is obtained, and if the area of the pixel region projected by the patch into the reference image is not larger than the preset value, the texture complexity of the corresponding region of the patch on the reference image is not obtained. And if the areas of the pixel regions respectively projected to the N reference images by the patch are all larger than a preset value, the M is equal to N. And if the areas of the pixel regions respectively projected to the N reference images by the patch are not all larger than a preset value, the M is smaller than the N. Wherein, the preset value is determined according to the area of the patch.

In the following, one patch is taken as an example, and the processing of the other patches is the same. After the texture complexity of the corresponding areas of the patches on the M reference images is obtained, whether the patches are to-be-optimized patches is determined according to the texture complexity of the corresponding areas of the patches on the M reference images. And if the patch is determined to be the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model, wherein the patch after the position adjustment is the optimized patch so as to obtain the optimized three-dimensional reconstruction model. If the patch is determined not to be the patch to be optimized, the position of the patch in the three-dimensional reconstruction model is not required to be adjusted, namely the position of the patch in the three-dimensional reconstruction model is kept unchanged.

In the optimization method of the three-dimensional reconstruction model provided by this embodiment, the texture complexity of the corresponding region of each patch in the three-dimensional reconstruction model to be optimized of the target object on the M reference images is obtained; determining whether the patch is a patch to be optimized according to the texture complexity of the corresponding areas of the patch on the M reference images; and if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model. Because the embodiment does not directly optimize all patches, but optimizes the determined patches to be optimized, the optimization efficiency of the three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced.

In some embodiments, one possible implementation manner of the foregoing S302 may include S3021 and S3022:

s3021, determining the target texture complexity of the patch according to the texture complexity of the corresponding areas of the patch on the M reference images.

And S3022, determining whether the patch is a patch to be optimized according to the comparison result of the target texture complexity and a preset value.

In this embodiment, according to the texture complexity of the corresponding region of the patch on the M reference images, a target texture complexity of the patch may be determined, where the target texture complexity includes, for example, one of the following: the maximum texture complexity of the patch in the M reference images, the minimum texture complexity of the patch in the M reference images, and the average texture complexity of the patch in the M reference images. It should be noted that the target texture complexity of the present embodiment is not limited thereto. Then, according to the target re-projection error, determining whether the patch is a patch to be optimized, wherein one implementation manner may be: according to the comparison result of the target texture complexity and the preset texture complexity, determining whether the patch is a patch to be optimized, for example: and judging whether the target texture complexity of the surface patch is greater than the preset texture complexity, if so, determining that the surface patch is the surface patch to be optimized, and if not, determining that the surface patch is not the surface patch to be optimized.

According to the texture complexity of the patch projected to M reference images respectively, the target texture complexity of the patch is determined, and whether the patch is a patch to be optimized is determined according to the target texture complexity, so that the patch to be optimized can be determined more accurately, the optimization efficiency of a three-dimensional reconstruction model is improved, and the probability of noise introduction is reduced. For example: if the target texture complexity is the maximum texture complexity of the patch in the M reference images, the larger the maximum texture complexity is, the more the patch should be optimized.

In some embodiments, based on the above embodiments, one possible implementation manner of adjusting the positions of the patches in the three-dimensional reconstruction model in the above S203 and S303 to obtain the optimized three-dimensional reconstruction model may include steps a1 and a 2:

and A1, dividing the patch into K patches.

And A2, adjusting the position of each patch in the K patches in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

In this embodiment, after the patch is determined to be a patch to be converted into a patch, the patch is subdivided, that is, the patch is divided into K patches, where K is an integer greater than or equal to 2. And then adjusting the position of each of the K patches in the three-dimensional reconstruction model to obtain an optimized three-dimensional reconstruction model, wherein the optimized three-dimensional reconstruction model comprises the K patches with the adjusted positions.

In this embodiment, the patch is subdivided into smaller patches, and the smaller patches are optimized, so that the optimized three-dimensional reconstruction model has finer details.

Optionally, the area of each of the K patches projected to the pixel region in the P reference images is smaller than or equal to a preset area. The method can obtain K patches which are respectively projected to the areas of the pixel regions in the P reference images and have the area smaller than or equal to the preset area by thinning the patch to be optimized once, and also can obtain the K patches by thinning the patch to be optimized for multiple times.

In some embodiments, one possible implementation manner of the above a2 is: dividing the surface patches into H surface patches, wherein H is an integer which is more than or equal to 2 and less than or equal to K; if the areas of the divided patches projected to the pixel regions in the P reference images are larger than the preset area, the divided patches are divided into H patches again until the area of each patch projected to the pixel regions in the P reference images is smaller than or equal to the preset area.

Firstly, dividing a surface patch into H surface patches, wherein H is an integer which is more than or equal to 2 and less than or equal to K; judging whether the area of each of H patches obtained after the first division, which is projected to the pixel region in the P reference images respectively, is larger than a preset area or not, and if the area of at least one patch obtained after the first division, which is projected to the pixel region in the P reference images respectively, is larger than the preset area, dividing the at least one patch obtained after the first division into H patches respectively; and if the area of each patch after the first division projected to the pixel region in the P reference images is smaller than or equal to the preset area, the patches are not continuously divided. Judging whether the area of each patch of H patches obtained after the second division, which is respectively projected to the pixel regions in the P reference images, is larger than the preset area, and if the area of each patch which is respectively projected to the pixel regions in the P reference images after the second division is smaller than or equal to the preset area, not continuously dividing the patches for the patches; if the area of the at least one patch projected to the pixel regions in the P reference images after the second division is larger than the preset area, dividing each patch in the at least one patch after the second division into H patches again until the area of each patch projected to the pixel regions in the P reference images after the division of the K patches is smaller than or equal to the preset area.

It should be noted that, in some cases, after each patch is divided into H patches for the first time, K patches may be obtained, and the area of each patch of the K patches projected to the pixel regions in P reference images is smaller than or equal to a preset area.

In some embodiments, one possible implementation manner of dividing the patch into H patches is as follows: and adding a vertex on each edge in the patch, and sequentially connecting the newly added vertices to form edges so as to obtain H patches. Taking a patch as an example of a triangle patch, as shown in fig. 4, for example, H is equal to 4, a vertex D is added to a side (for example, the middle position of the side) between vertices a and B of the triangle patch, a vertex E is added to a side (for example, the middle position of the side) between vertices B and C of the triangle patch, a vertex F is added to a side (for example, the middle position of the side) between vertices a and C of the triangle patch, the vertex D and the vertex E are connected to form a side, the vertex D and the vertex F are connected to form a side, and the vertex E and the vertex F are connected to form a side, thereby dividing the original triangle patch into four triangle patches.

In some embodiments, on the basis of the foregoing embodiments, after performing the foregoing S203 or S303, the three-dimensional reconstruction model optimized at the previous downsampling level of the initial image is updated to the three-dimensional reconstruction model to be optimized of the target object at the next downsampling level of the initial image.

Taking the initial value of L equal to 2 as an example, for L equal to 2, performing the above S201-S203 or S301-S303, and obtaining a three-dimensional reconstruction model after the initial image is optimized when down-sampled by 2 levels. Then, updating L to 1, and performing the above S201-S203 or S301-S303 again for L to 1, wherein in the step of performing S201 or S301, the three-dimensional reconstruction model to be optimized of the target object is the three-dimensional reconstruction model optimized in the step of down-sampling 2 levels. When the L is again updated to 0, S201 to S203 or S301 to S303 are executed again with respect to L being 0, and when S201 or S301 is executed, the three-dimensional reconstruction model to be optimized of the target object is the three-dimensional reconstruction model optimized when the down-sampling is 1 level, it should be noted that when L is 0, the reference image is the initial image.

And the three-dimensional reconstruction model obtained by current optimization is optimized once again every time L is updated, so that the accuracy of the three-dimensional reconstruction model obtained finally is higher. And each time L is updated, the resolution of the reference image is increased, the patch is reduced, and accordingly the precision of the obtained optimized three-dimensional reconstruction model is improved.

It should be noted that the above-mentioned embodiment shown in fig. 2 and the embodiment shown in fig. 3 may be used in combination, for example, fig. 2 and fig. 3 are used alternately, when the three-dimensional reconstruction model is optimized for the first time, the embodiment shown in fig. 2 is adopted, L is updated once and the embodiment shown in fig. 3 is adopted, L is updated once again and the embodiment shown in fig. 2 is adopted, and so on. It should be noted that the example of using fig. 2 and 3 in combination is not limited to this.

In some embodiments, based on the above embodiments, one possible implementation manner of adjusting the position of the patch in the three-dimensional reconstruction model in the above embodiments may be:

and obtaining the position of the patch after the ith position adjustment, and adjusting the position of the patch in the three-dimensional reconstruction model again, wherein i is an integer greater than or equal to 0.

In this embodiment, the position of the patch in the three-dimensional reconstruction model needs to be adjusted for many times, the position of the patch in the three-dimensional reconstruction model is adjusted for the first time, then the position of the patch adjusted for the first time in the three-dimensional reconstruction model is obtained, the position of the patch in the three-dimensional reconstruction model is adjusted for the second time on the basis of the position of the patch adjusted for the first time in the three-dimensional reconstruction model, then the position of the patch adjusted for the second time in the three-dimensional reconstruction model is obtained, the position of the patch in the three-dimensional reconstruction model is adjusted for the third time on the basis of the position of the patch adjusted for the second time in the three-dimensional reconstruction model, and so on until the number of times of adjustment is equal to. In the embodiment, the three-dimensional reconstruction model is optimized by gradually adjusting the positions of the patches in the three-dimensional reconstruction model for multiple times, so that the introduction of larger noise is avoided.

In some embodiments, one possible implementation manner of adjusting the position of the patch in the three-dimensional reconstruction model in the above embodiments is as follows: and adjusting the positions of the vertexes of the patches in the three-dimensional reconstruction model. The position of the vertex changes, and accordingly, the position of the patch changes.

In some embodiments, one possible implementation manner of adjusting the positions of the vertices of the patch in the three-dimensional reconstruction model is as follows: acquiring the gradient of the vertex of the patch; and adjusting the positions of the vertexes of the patch in the three-dimensional reconstruction model according to the gradients of the vertexes.

In this embodiment, the gradient of each vertex of the patch is obtained, and for each vertex, the position of the vertex of the patch in the three-dimensional reconstruction model is adjusted according to the gradient of the vertex. If the position of the patch needs to be adjusted for multiple times, acquiring the gradient of the vertex for the first time, adjusting the position of the vertex of the patch in the three-dimensional reconstruction model according to the gradient of the vertex acquired for the first time, acquiring the gradient of the vertex after the position is adjusted for the first time, adjusting the position of the vertex in the three-dimensional reconstruction model for the second time according to the gradient of the vertex after the position is adjusted for the first time, acquiring the gradient of the vertex after the position is adjusted for the second time, adjusting the position of the vertex in the three-dimensional reconstruction model for the third time according to the gradient of the vertex after the position is adjusted for the second time, and so on until the adjustment times are equal to the.

In some embodiments, one possible implementation manner of adjusting the positions of the vertices of the patch in the three-dimensional reconstruction model according to the gradients of the vertices is as follows: and moving the vertex of the patch by a preset step length along the direction corresponding to the gradient of the vertex. Optionally, the preset step size is related to the side lengths of the plurality of patches to which the vertex belongs. As shown in fig. 5, taking a patch as a triangle patch as an example, the vertex O belongs to the triangle patches 1 to 6, and the preset step length of the vertex O is related to the side lengths of the triangle patches 1 to 6, for example, the preset step length of the vertex O is related to the average side length of all the side lengths in the triangle patches 1 to 6, and is, for example, less than or equal to a multiple of the average side length of all the side lengths in the triangle patches 1 to 6, and the multiple is, for example, 0.2.

In some embodiments, one possible implementation manner of obtaining the gradient of the vertex of the patch is as follows: acquiring the gradient of each point in a plurality of patches to which the vertex belongs; and acquiring the gradient of the vertex according to the gradient of each point in a plurality of patches to which the vertex belongs.

As shown in fig. 5, the gradients of each point in triangle patch 1 to triangle patch 6 are obtained, and then the gradient of vertex O is obtained according to the gradients of these points.

In some embodiments, one possible implementation manner of obtaining the gradient of the vertex according to the gradient of each point in the plurality of patches to which the vertex belongs is as follows: and determining the gradient of the vertex according to the integral after the gradient weighting of each point in a plurality of patches to which the vertex belongs.

Taking fig. 5 as an example, the gradient of the vertex O is determined by an integral weighted by the gradients of each point in the triangle patch 1 to the triangle patch 6, wherein the weight of the gradient of each point is positively correlated with the distance between the point and the vertex O.

The gradient weighted integral of each point in the plurality of patches to which the vertex belongs can be represented by the following formula:

where N represents a set of patches to which the vertices belong (e.g., triangle patch 1 through triangle patch 6 as shown in fig. 5), x represents points in the patches to which the vertices belong,

denotes the gradient of point x, phi (x) being

The weight of (c). Where φ (x) is related to the position of each vertex of the patch to which point x belongs and the distance between point x and each vertex. Taking the patch as an example in fig. 5, assuming that the point x is a point in the triangle patch 1, the relationship between the point x and the three vertices of the triangle to which the point x belongs is: px is w 1P_A+w2*P_B+w3*P_C，P_AIndicates the position, P, of the vertex A of the triangular patch to which point x belongs_BIndicates the position of the vertex B, P, of the triangular patch to which the point x belongs_CRepresents the position of the vertex C of the triangle patch to which the point x belongs, and Px represents the position of the point x in the triangle patch to which the point x belongs. w1 relates to the distance between point x and vertex A, for example: the closer the distance between point x and vertex a, the larger w 1. w2 relates to the distance between point x and vertex B, for example: the closer the distance between point x and vertex B, the larger w 2. w3 relates to the distance between point x and vertex C, for example: dotThe closer the distance between x and the vertex C, the larger w 3. When the gradient of the vertex A needs to be obtained through the formula, phi (x) in the formula is w 1; when the gradient of the vertex B needs to be obtained through the formula, phi (x) in the formula is w 2; when the gradient of the vertex C needs to be obtained through the above formula, phi (x) in the above formula takes the value of w 3.

Fig. 6 is a schematic structural diagram of an optimization apparatus of a three-dimensional reconstruction model according to an embodiment of the present application, and as shown in fig. 6, the optimization apparatus 600 of a three-dimensional reconstruction model according to this embodiment may include: camera 601 and processor 602. The camera 601 and the processor 602 may be connected via a bus.

And the shooting device 601 is used for acquiring an initial image.

The processor 602 is configured to obtain a reprojection error that each patch in the to-be-optimized three-dimensional reconstruction model of the target object is projected into M reference images; determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively; and if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

The three-dimensional reconstruction model is generated by the processor 602 according to N initial images captured by the capturing device, and is composed of a plurality of patches, where the N initial images at least include partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 2 and less than or equal to N.

In some embodiments, the processor 602 is further configured to, after the determining whether the patch is a patch to be optimized, if the patch is not the patch to be optimized, keep a position of the patch in the three-dimensional reconstruction model unchanged.

In some embodiments, the processor 602 is further configured to identify the patch that is the patch to be optimized and/or the patch that is not the patch to be optimized.

In some embodiments, the processor 602 is specifically configured to:

determining gradients of regions of the patch re-projected on the M reference images according to the re-projection errors of the patch projected on the M reference images respectively;

and determining whether the patch is a patch to be optimized according to the gradients of the reprojected areas of the patch on the M reference images.

In some embodiments, the processor 602 is specifically configured to:

determining the target gradient of the patch according to the gradient of the reprojected areas of the patch on the M reference images respectively;

and determining whether the patch is a patch to be optimized according to the comparison result of the target gradient and a preset gradient.

In some embodiments, the target gradient comprises one of:

the maximum gradient of the gradients of the reprojected regions of the patch on the M reference images, the minimum gradient of the gradients of the reprojected regions of the patch on the M reference images, and the average gradient of the gradients of the reprojected regions of the patch on the M reference images.

In some embodiments, if the target gradient is greater than a preset gradient, the patch is a patch to be optimized.

In some embodiments, the processor 602 is specifically configured to:

determining a target reprojection error of the patch according to the reprojection errors of the patch projected to the M reference images respectively;

and determining whether the patch is a patch to be optimized according to the target reprojection error.

In some embodiments, the processor 602 is specifically configured to:

and determining whether the patch is a patch to be optimized according to the comparison result of the target reprojection error and a preset reprojection error.

In some embodiments, the target reprojection error comprises one of:

the maximum reprojection error of the patch in the M reference images, the minimum reprojection error of the patch in the M reference images, and the average reprojection error of the patch in the M reference images.

In some embodiments, if the target reprojection error is greater than a preset reprojection error, the patch is a patch to be optimized.

In some embodiments, the processor 602 is specifically configured to:

dividing the surface patches into K surface patches, wherein K is an integer greater than or equal to 2;

adjusting the position of each patch in the K patches in the three-dimensional reconstruction model to obtain an optimized three-dimensional reconstruction model;

wherein the optimized three-dimensional reconstruction model comprises the K patches of adjusted positions.

In some embodiments, the area of each of the K directional slices for projecting the pixel regions in all P reference images to the slice is less than or equal to a preset area.

In some embodiments, the processor 602 is specifically configured to:

dividing the surface patches into H surface patches, wherein H is an integer which is more than or equal to 2 and less than or equal to K;

if the areas of the divided patches projected to the pixel regions in the P reference images are larger than the preset area, the divided patches are divided into H patches again until the area of each patch projected to the pixel regions in the P reference images is smaller than or equal to the preset area.

In some embodiments, the processor 602 is specifically configured to:

and adding a vertex on each edge in the patch, and sequentially connecting the newly added vertices to form edges so as to obtain H patches.

In some embodiments, the processor 602 is further configured to update the three-dimensional reconstruction model optimized at the previous downsampling level of the initial image to the three-dimensional reconstruction model to be optimized of the target object at the next downsampling level of the initial image.

In some embodiments, the processor 602 is specifically configured to:

and adjusting the positions of the vertexes of the patch in the three-dimensional reconstruction model.

In some embodiments, the processor 602 is specifically configured to:

acquiring the gradient of the vertex of the patch;

and adjusting the positions of the vertexes of the patch in the three-dimensional reconstruction model according to the gradients of the vertexes.

In some embodiments, the processor 602 is specifically configured to:

and moving the vertex of the patch by a preset step length along the direction corresponding to the gradient of the vertex.

In some embodiments, the preset step size is related to a side length of a plurality of patches to which the vertex belongs.

In some embodiments, the processor 602 is specifically configured to:

acquiring the gradient of each point in a plurality of patches to which the vertex belongs;

and acquiring the gradient of the vertex according to the gradient of each point in a plurality of patches to which the vertex belongs.

In some embodiments, the processor 602 is specifically configured to:

and determining the gradient of the vertex according to the integral after the gradient weighting of each point in a plurality of patches to which the vertex belongs.

In some embodiments, the patches are triangular patches.

Optionally, the apparatus 600 for optimizing a three-dimensional reconstruction model according to this embodiment may further include: a memory (not shown) for storing program code, the memory being configured to store program code, and the optimization apparatus 600 for three-dimensional reconstruction model can implement the above-mentioned solution when the program code is executed.

The optimization device of the three-dimensional reconstruction model of this embodiment may be used to execute the technical solution of fig. 2 and the corresponding method embodiment, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 7 is a schematic structural diagram of an optimization apparatus for a three-dimensional reconstruction model according to another embodiment of the present application, and as shown in fig. 7, an optimization apparatus 700 for a three-dimensional reconstruction model according to this embodiment may include: a camera 701 and a processor 702. The camera 701 and the processor 702 may be connected by a bus.

And the shooting device 701 is used for acquiring an initial image.

A processor 702, configured to obtain texture complexity of a corresponding region of each patch in the to-be-optimized three-dimensional reconstruction model of the target object on the M reference images; determining whether the patch is a patch to be optimized according to the texture complexity of the patch in the corresponding area on the M reference images; and if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

The three-dimensional reconstruction model is generated by the processor 702 according to N initial images acquired by the shooting device 701 and is composed of a plurality of patches, the N initial images at least include partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 2, and M is an integer greater than or equal to 2 and less than or equal to N.

In some embodiments, the processor 702 is further configured to, after the determining whether the patch is a patch to be optimized, if the patch is not the patch to be optimized, keep a position of the patch in the three-dimensional reconstruction model unchanged.

In some embodiments, the processor 702 is further configured to:

identifying the patches that are the patches to be optimized and/or not the patches to be optimized.

In some embodiments, the processor 702 is specifically configured to:

determining the target texture complexity of the patch according to the texture complexity of the patch in the corresponding area of the M reference images;

and determining whether the patch is a patch to be optimized according to the comparison result of the target texture complexity and a preset value.

In some embodiments, the target texture complexity comprises one of:

a maximum texture complexity of texture complexities of corresponding regions of the patch over the M reference images, a minimum texture complexity of texture complexities of corresponding regions of the patch over the M reference images, and an average texture complexity of texture complexities of corresponding regions of the patch over the M reference images.

In some embodiments, if the target texture complexity is greater than a preset texture complexity, the patch is a patch to be optimized.

In some embodiments, the processor 702 is specifically configured to:

In some embodiments, the area of each of the K patches projected to the pixel regions in the P reference images is smaller than or equal to a preset area.

In some embodiments, the processor 702 is specifically configured to:

In some embodiments, the processor 702 is further configured to update the three-dimensional reconstruction model optimized at the previous downsampling level of the initial image to the three-dimensional reconstruction model to be optimized of the target object at the next downsampling level of the initial image.

In some embodiments, the processor 702 is specifically configured to:

acquiring the gradient of the vertex of the patch;

In some embodiments, the processor 702 is specifically configured to:

In some embodiments, the patches are triangular patches.

Optionally, the apparatus for optimizing a three-dimensional reconstruction model 700 of the present embodiment may further include: a memory (not shown) for storing program code, the memory being used for storing program code, the optimization apparatus 700 for three-dimensional reconstruction model being capable of implementing the above-mentioned solution when the program code is executed.

The optimization device of the three-dimensional reconstruction model of this embodiment may be used to execute the technical solution of fig. 3 and the corresponding method embodiment, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 8 is a schematic structural diagram of a movable platform according to an embodiment of the present application, and as shown in fig. 8, the movable platform 800 according to the embodiment may include: a camera 801 and a processor 802. The camera 801 and the processor 802 may be connected by a bus.

And the shooting device 801 is used for acquiring an initial image.

The processor 802 is configured to obtain a reprojection error of each patch in the to-be-optimized three-dimensional reconstruction model of the target object projected into M reference images; determining whether the patch is a patch to be optimized according to the reprojection errors of the patch projected to the M reference images respectively; and if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

The three-dimensional reconstruction model is generated by the processor 802 according to N initial images captured by the capturing device 801, and is a model composed of a plurality of patches, where the N initial images at least include partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 3, and M is an integer greater than or equal to 2 and less than or equal to N.

In some embodiments, the processor 802 is further configured to, after the determining whether the patch is a patch to be optimized, if the patch is not the patch to be optimized, keep a position of the patch in the three-dimensional reconstruction model unchanged.

In some embodiments, the processor 802 is further configured to identify the patch that is the patch to be optimized and/or the patch that is not the patch to be optimized.

In some embodiments, the processor 802 is specifically configured to:

In some embodiments, the target gradient comprises one of:

In some embodiments, the processor 802 is specifically configured to:

In some embodiments, the target reprojection error comprises one of:

In some embodiments, the processor 802 is specifically configured to:

In some embodiments, the processor 802 is further configured to update the three-dimensional reconstruction model optimized at the previous downsampling level of the initial image to the three-dimensional reconstruction model to be optimized of the target object at the next downsampling level of the initial image.

In some embodiments, the processor 802 is specifically configured to:

acquiring the gradient of the vertex of the patch;

In some embodiments, the processor 802 is specifically configured to:

In some embodiments, the patches are triangular patches.

Optionally, the movable platform 800 of this embodiment may further include: a memory (not shown) for storing program code, the memory being configured to store program code, the removable platform 800 being configured to implement the above-described aspects when the program code is executed.

The movable platform of this embodiment may be used to implement the technical solutions of fig. 2 and the corresponding method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again.

Fig. 9 is a schematic structural diagram of a movable platform according to another embodiment of the present application, and as shown in fig. 9, the movable platform 900 according to this embodiment may include: a camera 901 and a processor 902. The camera 901 and the processor 902 may be connected by a bus.

And the shooting device 901 is used for acquiring an initial image.

A processor 902, configured to obtain texture complexity of each patch in the to-be-optimized three-dimensional reconstruction model of the target object in a corresponding region on the M reference images; determining whether the patch is a patch to be optimized according to the texture complexity of the patch in the corresponding area on the M reference images; and if the patch is the patch to be optimized, adjusting the position of the patch in the three-dimensional reconstruction model to obtain the optimized three-dimensional reconstruction model.

The three-dimensional reconstruction model is generated by the processor 902 according to N initial images collected by the shooting device 901, and is a model composed of a plurality of patches, where the N initial images at least include partial images of the target object, the reference image is an image obtained by downsampling the initial images by L levels, L is an integer greater than or equal to 0, N is an integer greater than or equal to 2, and M is an integer greater than or equal to 2 and less than or equal to N.

In some embodiments, the processor 902 is further configured to, after the determining whether the patch is a patch to be optimized, if the patch is not the patch to be optimized, keep a position of the patch in the three-dimensional reconstruction model unchanged.

In some embodiments, the processor 902 is further configured to:

In some embodiments, the processor 902 is specifically configured to:

In some embodiments, the target texture complexity comprises one of:

In some embodiments, the processor 902 is specifically configured to:

In some embodiments, the processor 902 is further configured to update the three-dimensional reconstructed model optimized at the previous downsampling level of the initial image to the three-dimensional reconstructed model to be optimized of the target object at the next downsampling level of the initial image.

In some embodiments, the processor 902 is specifically configured to:

acquiring the gradient of the vertex of the patch;

In some embodiments, the processor 902 is specifically configured to:

In some embodiments, the patches are triangular patches.

Optionally, the movable platform 900 of the present embodiment may further include: a memory (not shown) for storing program code, the memory for storing program code, the removable platform 900 when executed, may implement the above-described aspects.

The movable platform of this embodiment may be used to implement the technical solutions of fig. 3 and the corresponding method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again.

Fig. 10 is a schematic structural diagram of a movable platform according to another embodiment of the present application, and as shown in fig. 10, the movable platform 1000 according to this embodiment may include: a movable platform body 1001 and a three-dimensional reconstruction model optimization apparatus 1002.

Wherein the optimization apparatus 1002 of the three-dimensional reconstruction model is mounted on the movable platform body 1001. The optimization apparatus 1002 of the three-dimensional reconstruction model may be an apparatus independent of the movable platform body 1001.

The optimization device 1002 of the three-dimensional reconstruction model may adopt the structure of the apparatus embodiment shown in fig. 6, and accordingly, may execute the technical solutions of fig. 2 and the corresponding method embodiments, and the implementation principles and technical effects thereof are similar, and are not described herein again. Alternatively, the first and second electrodes may be,

the optimization device 1002 of the three-dimensional reconstruction model may adopt the structure of the apparatus embodiment shown in fig. 7, and accordingly, may execute the technical solutions of fig. 3 and the corresponding method embodiments, which have similar implementation principles and technical effects, and are not described herein again.

Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: various media capable of storing program codes, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing a three-dimensional reconstruction model, comprising:

2. The method of claim 1, wherein after the determining whether the patch is a patch to be optimized, the method further comprises:

and if the patch is not the patch to be optimized, keeping the position of the patch in the three-dimensional reconstruction model unchanged.

3. The method of claim 2, further comprising:

4. The method according to any one of claims 1 to 3, wherein the determining whether the patch is a patch to be optimized according to the reprojection errors of the patch respectively projected into the M reference images comprises:

5. The method of claim 4, wherein the determining whether the patch is a patch to be optimized according to the gradients of the regions of the re-projection of the patch on the M reference images respectively comprises:

6. The method of claim 5, wherein the target gradient comprises one of:

7. The method of claim 5 or 6, wherein the patch is a patch to be optimized if the target gradient is greater than a preset gradient.

8. The method according to any one of claims 1 to 3, wherein the determining whether the patch is a patch to be optimized according to the reprojection errors of the patch respectively projected into the M reference images comprises:

9. The method of claim 8, wherein determining whether the patch is a patch to be optimized according to the target reprojection error comprises:

10. The method of claim 9, wherein the target reprojection error comprises one of:

11. The method according to claim 9 or 10, wherein the patch is a patch to be optimized if the target reprojection error is greater than a preset reprojection error.

12. The method according to any one of claims 1-11, wherein the adjusting the position of the patch in the three-dimensional reconstructed model to obtain an optimized three-dimensional reconstructed model comprises:

13. The method of claim 12, wherein the area of each of the K patches projected to the pixel region in the P reference images is smaller than or equal to a predetermined area.

14. The method of claim 13, wherein the dividing the patch into K patches comprises:

if the areas of the divided patches projected to the pixel regions in the P reference images are larger than the preset area, the divided patches are divided into H patches again until the area of each of the K divided patches projected to the pixel regions in the P reference images is smaller than or equal to the preset area.

15. The method of claim 14, wherein the dividing the patch into H patches comprises:

16. The method according to any one of claims 1-15, further comprising:

and updating the L to be equal to L-1, and updating the three-dimensional reconstruction model after the initial image is optimized at the previous sampling level and the three-dimensional reconstruction model to be optimized of the target object when the initial image is at the next sampling level.

17. The method of any of claims 1-16, wherein adjusting the position of the patch in the three-dimensional reconstructed model comprises:

18. The method of any of claims 1-17, wherein the adjusting the position of the patch in the three-dimensional reconstructed model comprises:

19. The method of claim 18, wherein the adjusting the locations of the vertices of the patch in the three-dimensional reconstructed model comprises:

acquiring the gradient of the vertex of the patch;

20. The method of claim 19, wherein the adjusting the positions of the vertices of the patch in the three-dimensional reconstructed model according to the gradients of the vertices comprises:

21. The method of claim 20, wherein the preset step size is related to a side length of a plurality of patches to which the vertex belongs.

22. The method of any of claims 19-21, wherein obtaining the gradient of the vertices of the patch comprises:

23. The method of claim 22, wherein obtaining the gradient of the vertex from the gradient of each point in the plurality of patches to which the vertex belongs comprises:

24. The method of any of claims 1-23, wherein the patch is a triangular patch.

25. A method for optimizing a three-dimensional reconstruction model, comprising:

26. The method of claim 25, wherein after the determining whether the patch is a patch to be optimized, the method further comprises:

27. The method of claim 26, further comprising:

28. The method according to any one of claims 25 to 27, wherein the determining whether the patch is a patch to be optimized according to the texture complexity of the corresponding region of the patch on M reference images, respectively, comprises:

29. The method of claim 28, wherein the target texture complexity comprises one of:

30. The method of claim 28 or 29, wherein the patch is a patch to be optimized if the target texture complexity is greater than a preset texture complexity.

31. The method of any one of claims 25-30, wherein the adjusting the position of the patch in the three-dimensional reconstructed model to obtain an optimized three-dimensional reconstructed model comprises:

32. The method of claim 31, wherein the area of each of the K patches projected to the pixel region in the P reference images is smaller than or equal to a predetermined area.

33. The method of claim 32, wherein the dividing the patch into K patches comprises:

34. The method of claim 33, wherein the dividing the patch into H patches comprises:

35. The method according to any one of claims 25-34, further comprising:

36. The method of any of claims 25-35, wherein adjusting the position of the patch in the three-dimensional reconstructed model comprises:

37. The method of any of claims 25-36, wherein the adjusting the position of the patch in the three-dimensional reconstructed model comprises:

38. The method of claim 37, wherein the adjusting the positions of the vertices of the patch in the three-dimensional reconstructed model comprises:

acquiring the gradient of the vertex of the patch;

39. The method of claim 38, wherein the adjusting the positions of the vertices of the patch in the three-dimensional reconstructed model according to the gradients of the vertices comprises:

40. The method of claim 39, wherein the preset step size is related to a side length of a plurality of patches to which the vertex belongs.

41. The method of any of claims 38-40, wherein obtaining the gradient of the vertices of the patch comprises:

42. The method of claim 41, wherein obtaining the gradient of the vertex from the gradient of each point in the plurality of patches to which the vertex belongs comprises:

and determining the integral after weighting the gradient of each point in a plurality of patches to which the vertex belongs as the gradient of the vertex.

43. The method of any of claims 25-42, wherein the patch is a triangular patch.

44. An apparatus for optimizing a three-dimensional reconstruction model, comprising:

the shooting device is used for acquiring an initial image;

45. The apparatus of claim 44, wherein the processor is further configured to, after the determining whether the patch is a patch to be optimized, if the patch is not the patch to be optimized, keep a position of the patch in the three-dimensional reconstruction model unchanged.

46. The apparatus of claim 45, wherein the processor is further configured to identify the patches that are the patches to be optimized and/or not the patches to be optimized.

47. The device according to any one of claims 44 to 46, wherein the processor is specifically configured to:

48. The device of claim 47, wherein the processor is specifically configured to:

49. The apparatus of claim 48, wherein the target gradient comprises one of:

50. The apparatus of claim 48 or 49, wherein the patch is a patch to be optimized if the target gradient is greater than a preset gradient.

51. The device according to any one of claims 44 to 46, wherein the processor is specifically configured to:

52. The device of claim 51, wherein the processor is specifically configured to:

53. The apparatus of claim 52, wherein the target reprojection error comprises one of:

54. The apparatus of claim 52 or 53, wherein the patch is a patch to be optimized if the target reprojection error is greater than a preset reprojection error.

55. The device according to any of claims 44-54, wherein the processor is specifically configured to:

56. The apparatus of claim 55, wherein the area of each of the K patches projected to the pixel region in the P reference images is smaller than or equal to a predetermined area.

57. The device of claim 56, wherein the processor is specifically configured to:

58. The device of claim 57, wherein the processor is specifically configured to:

59. The apparatus according to any of claims 44-58, wherein the processor is further configured to update the three-dimensional reconstructed model optimized for the initial image at a previous downsampling level to a three-dimensional reconstructed model to be optimized for the target object at a subsequent downsampling level for the initial image, wherein L is equal to L-1.

60. The device according to any one of claims 44 to 59, wherein the processor is specifically configured to:

61. The device according to any of claims 44-60, wherein the processor is specifically configured to:

62. The device according to claim 61, wherein the processor is specifically configured to:

acquiring the gradient of the vertex of the patch;

63. The device of claim 62, wherein the processor is specifically configured to:

64. The apparatus of claim 63, wherein the preset step size is related to a side length of a plurality of patches to which the vertex belongs.

65. The device according to any one of claims 62 to 64, wherein the processor is specifically configured to:

66. The device of claim 65, wherein the processor is specifically configured to:

67. The apparatus of any one of claims 44-66, wherein the patches are triangular patches.

68. An apparatus for optimizing a three-dimensional reconstruction model, comprising:

the shooting device is used for acquiring an initial image;

69. The apparatus of claim 68, wherein the processor is further configured to, after the determining whether the patch is a patch to be optimized, if the patch is not the patch to be optimized, keep a position of the patch in the three-dimensional reconstruction model unchanged.

70. The device of claim 69, wherein the processor is further configured to:

71. The device according to any one of claims 68-70, wherein the processor is specifically configured to:

72. The device of claim 71, wherein the target texture complexity comprises one of:

73. The apparatus of claim 71 or 72, wherein the patch is a patch to be optimized if the target texture complexity is greater than a preset texture complexity.

74. The device according to any one of claims 68-73, wherein the processor is specifically configured to:

75. The apparatus of claim 74, wherein the area of each of the K patches projected to the pixel region in the P reference images is smaller than or equal to a predetermined area.

76. The device of claim 75, wherein the processor is specifically configured to:

77. The device according to claim 76, wherein the processor is specifically configured to:

78. The device of any one of claims 68-77, wherein the processor is further configured to:

79. The device according to any one of claims 68-78, wherein the processor is specifically configured to:

80. The device of any one of claims 68-79, wherein the processor is specifically configured to:

81. The device of claim 80, wherein the processor is specifically configured to:

acquiring the gradient of the vertex of the patch;

82. The device of claim 81, wherein the processor is specifically configured to:

83. The apparatus of claim 82, wherein the preset step size is related to a side length of a plurality of patches to which the vertex belongs.

84. The device according to any one of claims 81-83, wherein the processor is specifically configured to:

85. The device according to claim 84, wherein the processor is specifically configured to:

86. The apparatus of any one of claims 68-85, wherein the patches are triangular patches.

87. A movable platform, comprising: a movable platform body and the optimization apparatus of the three-dimensional reconstruction model of any one of claims 44-86, wherein the optimization apparatus of the three-dimensional reconstruction model is mounted on the movable platform body.

88. The movable platform of claim 87, wherein the movable platform comprises a handheld phone, a handheld pan-tilt, a drone, an unmanned vehicle, an unmanned boat, a robot, or an autonomous automobile.

89. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program; the computer program, when executed, implements a method of optimizing a three-dimensional reconstruction model according to any one of claims 1-43.