CN111161336A - Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium - Google Patents

Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium Download PDF

Info

Publication number
CN111161336A
CN111161336A CN201911312900.9A CN201911312900A CN111161336A CN 111161336 A CN111161336 A CN 111161336A CN 201911312900 A CN201911312900 A CN 201911312900A CN 111161336 A CN111161336 A CN 111161336A
Authority
CN
China
Prior art keywords
geometric
detection
dimensional space
dimensional
straight line
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911312900.9A
Other languages
Chinese (zh)
Other versions
CN111161336B (en
Inventor
不公告发明人
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Urban Network Neighbor Information Technology Co Ltd
Original Assignee
Beijing Urban Network Neighbor Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Urban Network Neighbor Information Technology Co Ltd filed Critical Beijing Urban Network Neighbor Information Technology Co Ltd
Priority to CN201911312900.9A priority Critical patent/CN111161336B/en
Publication of CN111161336A publication Critical patent/CN111161336A/en
Application granted granted Critical
Publication of CN111161336B publication Critical patent/CN111161336B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/536—Depth or shape recovery from perspective effects, e.g. by using vanishing points
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Abstract

The embodiment of the disclosure provides a three-dimensional reconstruction method and a three-dimensional reconstruction device for a three-dimensional space and a computer readable storage medium. The three-dimensional reconstruction method of the three-dimensional space according to the embodiment of the disclosure comprises the following steps: respectively carrying out geometric shape detection and geometric relation detection on a plurality of panoramic images of the three-dimensional space by utilizing the geometric prior of the three-dimensional space, and determining a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction; according to the geometric prior of the three-dimensional space, constructing geometric constraints for the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection, and determining translation vectors and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints; performing a three-dimensional reconstruction of the three-dimensional space using the rotation matrix, the translation vector, and the position information.

Description

Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium
Technical Field
Embodiments of the present disclosure relate to the field of image processing, and in particular, to a three-dimensional reconstruction method, a three-dimensional reconstruction apparatus, and a computer-readable storage medium.
Background
Three-dimensional reconstruction refers to restoring a three-dimensional model of a three-dimensional object from a two-dimensional image, and computer three-dimensional reconstruction is an important research field in computer vision and computer graphics. There is a need for a method and apparatus for three-dimensional reconstruction of building interior space that is effective to more accurately and efficiently obtain the three-dimensional structure of the building interior space.
Disclosure of Invention
To solve the above technical problem, according to an aspect of the present disclosure, there is provided a three-dimensional reconstruction method of a three-dimensional space, including: respectively carrying out geometric shape detection and geometric relation detection on a plurality of panoramic images of the three-dimensional space by utilizing the geometric prior of the three-dimensional space, and determining a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction; according to the geometric prior of the three-dimensional space, constructing geometric constraints for the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection, and determining translation vectors and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints; performing a three-dimensional reconstruction of the three-dimensional space using the rotation matrix, the translation vector, and the position information.
According to some embodiments of the disclosure, wherein the geometric prior of the three-dimensional space comprises at least one of: two opposite surfaces in the three-dimensional space are parallel to each other, two adjacent surfaces in the three-dimensional space are perpendicular to each other, and three surfaces intersecting at a point in the three-dimensional space are perpendicular to each other pairwise.
According to some embodiments of the present disclosure, wherein the performing the geometry detection and the geometry relation detection on the plurality of panoramic images of the three-dimensional space respectively by using the geometry prior of the three-dimensional space comprises: correcting the panoramic image into a perspective image; performing geometry detection on the perspective image, wherein the geometry detection comprises straight line detection; and determining at least one vanishing point in the three-dimensional space at least according to the result of the straight line detection.
According to some embodiments of the present disclosure, the performing geometry detection and geometry relation detection on the plurality of panoramic images of the three-dimensional space respectively by using a geometric prior of the three-dimensional space, and determining a rotation matrix of a camera for three-dimensional reconstruction, the camera capturing the panoramic images, comprises: determining rotation matrixes of cameras corresponding to at least two perspective images according to the straight line detection results and the determined vanishing points of the at least two perspective images in the plurality of perspective images; or correcting the initial rotation matrix obtained by the inertial measurement unit of the camera according to the straight line detection result and the determined vanishing point of at least one perspective image in the plurality of perspective images to obtain the rotation matrix of the camera corresponding to the at least one perspective image.
According to some embodiments of the present disclosure, wherein constructing geometric constraints for the geometry and the geometric relationship detected in the geometry detection and the geometric relationship detection, respectively, according to a geometric prior of the three-dimensional space, and determining translation vectors and position information of a camera capturing the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints comprises: respectively carrying out feature point detection on a plurality of perspective images, carrying out feature point matching according to the result of the feature point detection, and acquiring at least one straight line on at least two perspective images in the plurality of perspective images; constructing a co-linear constraint for the at least one straight line, determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction at least from the constructed co-linear constraint.
According to some embodiments of the present disclosure, the performing geometry detection and geometry relation detection on the plurality of panoramic images of the three-dimensional space respectively by using a geometric prior of the three-dimensional space further includes: and performing the geometric relationship detection at least according to the result of the straight line detection, wherein the geometric relationship detection comprises at least one of the following steps: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection.
According to some embodiments of the present disclosure, wherein constructing geometric constraints for the geometry and the geometric relationship detected in the geometry detection and the geometric relationship detection, respectively, according to a geometric prior of the three-dimensional space, and determining translation vectors and position information of a camera capturing the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints comprises: respectively detecting feature points of a plurality of perspective images, and acquiring at least two straight lines which respectively exist on at least two perspective images of the plurality of perspective images and meet the detected geometric relationship; and constructing a geometric relation constraint aiming at the at least two straight lines, and determining a translation vector and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction at least according to the constructed geometric relation constraint.
According to some embodiments of the disclosure, the method further comprises: and according to the geometric prior of the three-dimensional space, carrying out bundle adjustment on the three-dimensional reconstruction result of the three-dimensional space.
According to some embodiments of the present disclosure, wherein the bundle adjustment of the result of the three-dimensional reconstruction of the three-dimensional space according to the geometric prior of the three-dimensional space comprises at least one of: parameterizing at least one straight line obtained in the three-dimensional reconstruction of the three-dimensional space, and performing bundle adjustment by utilizing a collinearity constraint aiming at the at least one straight line; parameterizing at least two coplanar straight lines obtained in the three-dimensional reconstruction of the three-dimensional space, and performing cluster adjustment by utilizing coplanarity constraint aiming at the at least two coplanar straight lines.
According to some embodiments of the present disclosure, the three-dimensional space is a building indoor space, and includes a bottom surface and a top surface parallel to each other and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form at least a partial enclosure.
According to another aspect of the present disclosure, there is provided a three-dimensional reconstruction apparatus of a three-dimensional space, including: a detection unit configured to perform geometry detection and geometry relation detection on a plurality of panoramic images of the three-dimensional space, respectively, using a geometric prior of the three-dimensional space, and determine a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction; a determination unit configured to construct geometric constraints for the geometric shapes and geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection according to a geometric prior of the three-dimensional space, and determine translation vectors and position information of cameras for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints; a reconstruction unit configured to perform a three-dimensional reconstruction of the three-dimensional space using the rotation matrix, the translation vector, and the position information.
According to some embodiments of the disclosure, wherein the geometric prior of the three-dimensional space comprises at least one of: two opposite surfaces in the three-dimensional space are parallel to each other, two adjacent surfaces in the three-dimensional space are perpendicular to each other, and three surfaces intersecting at a point in the three-dimensional space are perpendicular to each other pairwise.
According to some embodiments of the present disclosure, wherein the detection unit corrects the panoramic image to a perspective image; performing geometry detection on the perspective image, wherein the geometry detection comprises straight line detection; and determining at least one vanishing point in the three-dimensional space at least according to the result of the straight line detection.
According to some embodiments of the present disclosure, the detection unit determines rotation matrixes of cameras corresponding to at least two perspective images of a plurality of perspective images according to at least the straight line detection results and the determined vanishing point of the two perspective images; or correcting the initial rotation matrix obtained by the inertial measurement unit of the camera according to the straight line detection result and the determined vanishing point of at least one perspective image in the plurality of perspective images to obtain the rotation matrix of the camera corresponding to the at least one perspective image.
According to some embodiments of the present disclosure, the determining unit performs feature point detection on each of the plurality of perspective images, performs feature point matching according to a result of the feature point detection, and acquires at least one straight line on at least two perspective images of the plurality of perspective images; constructing a co-linear constraint for the at least one straight line, determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction at least from the constructed co-linear constraint.
According to some embodiments of the present disclosure, the detecting unit performs the geometric relationship detection at least according to a result of the line detection, wherein the geometric relationship detection includes at least one of: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection.
According to some embodiments of the present disclosure, the determining unit performs feature point detection on each of the plurality of perspective images, and obtains at least two straight lines, where the at least two straight lines exist on at least two of the plurality of perspective images, respectively, and satisfy the detected geometric relationship; and constructing a geometric relation constraint aiming at the at least two straight lines, and determining a translation vector and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction at least according to the constructed geometric relation constraint.
According to some embodiments of the disclosure, the apparatus further comprises: and the adjusting unit is configured to perform bundle adjustment on the result of the three-dimensional reconstruction of the three-dimensional space according to the geometric prior of the three-dimensional space.
According to some embodiments of the disclosure, the adjusting unit performs bundle adjustment by at least one of: parameterizing at least one straight line obtained in the three-dimensional reconstruction of the three-dimensional space, and performing bundle adjustment by utilizing a collinearity constraint aiming at the at least one straight line; parameterizing at least two coplanar straight lines obtained in the three-dimensional reconstruction of the three-dimensional space, and performing cluster adjustment by utilizing coplanarity constraint aiming at the at least two coplanar straight lines.
According to some embodiments of the present disclosure, the three-dimensional space is a building indoor space, and includes a bottom surface and a top surface parallel to each other and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form at least a partial enclosure.
According to another aspect of the present disclosure, there is provided a three-dimensional reconstruction apparatus of a three-dimensional space, including: one or more processors; and one or more memories having computer program instructions stored therein, wherein the computer program instructions, when executed by the one or more processors, perform the three-dimensional reconstruction method of any of the preceding claims.
According to another aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon instructions that, when executed by a processor, cause the processor to perform a three-dimensional reconstruction method as in any one of the preceding claims.
According to still another aspect of the present disclosure, there is provided a reconstruction method of a three-dimensional room, including: respectively carrying out geometric shape detection and geometric relation detection on a plurality of panoramic images of the three-dimensional room by utilizing the geometric prior of the three-dimensional room, and determining a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction; according to the geometric prior of the three-dimensional room, constructing geometric constraints for the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection, and determining translation vectors and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints; performing a three-dimensional reconstruction of the three-dimensional room using the rotation matrix, the translation vector, and the position information.
According to some embodiments of the disclosure, wherein the geometric prior of the three-dimensional room comprises at least one of: two opposite surfaces in the three-dimensional room are parallel to each other, two adjacent surfaces in the three-dimensional room are perpendicular to each other, and three surfaces intersecting at a point in the three-dimensional room are perpendicular to each other pairwise.
According to some embodiments of the present disclosure, wherein the performing the geometry detection and the geometry relation detection on the plurality of panoramic images of the three-dimensional room respectively using the geometry prior of the three-dimensional room comprises: correcting the panoramic image into a perspective image; performing geometry detection on the perspective image, wherein the geometry detection comprises straight line detection; determining at least one vanishing point in the three-dimensional room at least according to the result of the line detection.
According to some embodiments of the present disclosure, wherein the performing geometry detection and geometry relation detection on the plurality of panoramic images of the three-dimensional room respectively using the geometry prior of the three-dimensional room and determining a rotation matrix of a camera for three-dimensional reconstruction, the camera capturing the panoramic images, comprises: determining rotation matrixes of cameras corresponding to at least two perspective images according to the straight line detection results and the determined vanishing points of the at least two perspective images in the plurality of perspective images; or correcting the initial rotation matrix obtained by the inertial measurement unit of the camera according to the straight line detection result and the determined vanishing point of at least one perspective image in the plurality of perspective images to obtain the rotation matrix of the camera corresponding to the at least one perspective image.
According to some embodiments of the present disclosure, wherein constructing geometric constraints for the geometry and the geometric relation detected in the geometry detection and the geometric relation detection, respectively, according to a geometric prior of the three-dimensional room, and determining translation vectors and position information of a camera capturing the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints comprises: respectively carrying out feature point detection on a plurality of perspective images, carrying out feature point matching according to the result of the feature point detection, and acquiring at least one straight line on at least two perspective images in the plurality of perspective images; constructing a co-linear constraint for the at least one straight line, determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction at least from the constructed co-linear constraint.
According to some embodiments of the present disclosure, the method further comprises performing geometry detection and geometry relation detection on the plurality of panoramic images of the three-dimensional room respectively by using a geometric prior of the three-dimensional room, further comprising: and performing the geometric relationship detection at least according to the result of the straight line detection, wherein the geometric relationship detection comprises at least one of the following steps: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection.
According to some embodiments of the present disclosure, wherein constructing geometric constraints for the geometry and the geometric relation detected in the geometry detection and the geometric relation detection, respectively, according to a geometric prior of the three-dimensional room, and determining translation vectors and position information of a camera capturing the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints comprises: respectively detecting feature points of a plurality of perspective images, and acquiring at least two straight lines which respectively exist on at least two perspective images of the plurality of perspective images and meet the detected geometric relationship; and constructing a geometric relation constraint aiming at the at least two straight lines, and determining a translation vector and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction at least according to the constructed geometric relation constraint.
According to some embodiments of the disclosure, the method further comprises: and according to the geometric prior of the three-dimensional room, performing bundle adjustment on the three-dimensional reconstruction result of the three-dimensional room.
According to some embodiments of the present disclosure, wherein the bundle adjustment of the results of the three-dimensional reconstruction of the three-dimensional room according to the geometric prior of the three-dimensional room comprises at least one of: parameterizing at least one straight line obtained in the three-dimensional reconstruction of the three-dimensional room, and performing bundle adjustment by utilizing a collinearity constraint aiming at the at least one straight line; parameterizing at least two coplanar straight lines obtained in the three-dimensional reconstruction of the three-dimensional room, and performing bundle adjustment by utilizing coplanarity constraints aiming at the at least two coplanar straight lines.
According to some embodiments of the present disclosure, the three-dimensional room comprises a bottom surface and a top surface parallel to each other and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form an at least partial enclosure.
According to still another aspect of the present disclosure, there is provided a reconstruction apparatus of a three-dimensional room, including: a detection unit configured to perform geometry detection and geometry relation detection on a plurality of panoramic images of the three-dimensional room, respectively, using a geometric prior of the three-dimensional room, and determine a rotation matrix of a camera for capturing the panoramic images for three-dimensional reconstruction; a determination unit configured to construct geometric constraints for the geometric shapes and geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection according to a geometric prior of the three-dimensional room, and determine translation vectors and position information of cameras for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints; a reconstruction unit configured to perform a three-dimensional reconstruction of the three-dimensional room using the rotation matrix, the translation vector, and the position information.
According to still another aspect of the present disclosure, there is provided a reconstruction apparatus of a three-dimensional room, including: one or more processors; and one or more memories in which computer program instructions are stored, wherein the computer program instructions, when executed by the one or more processors, perform the method of reconstructing a three-dimensional room as in any one of the preceding claims.
According to another aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon instructions, which, when executed by a processor, cause the processor to perform a method of reconstructing a three-dimensional room as defined in any of the preceding claims.
According to the three-dimensional reconstruction method and the three-dimensional reconstruction device provided by the disclosure, the three-dimensional space, particularly the specific geometric prior of the internal space of a building, can be utilized to perform geometric shape detection and geometric relation detection on a panoramic image, and the panoramic image is used as a constraint condition to obtain various parameters required by three-dimensional reconstruction, so that the process of realizing the three-dimensional reconstruction is assisted. According to the three-dimensional reconstruction method and the three-dimensional reconstruction device, three-dimensional reconstruction can be accurately and efficiently carried out, and therefore the accurate three-dimensional internal structure of the internal space of the building can be obtained.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments of the present disclosure will be briefly introduced below, and it is apparent that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without inventive efforts.
Fig. 1 shows a flow chart of a method of three-dimensional reconstruction of a three-dimensional space according to an embodiment of the present disclosure;
FIG. 2 illustrates an example of a three-dimensional house view constructed by a three-dimensional reconstruction method according to an embodiment of the present disclosure;
fig. 3 shows a block diagram of a three-dimensional reconstruction apparatus of a three-dimensional space according to an embodiment of the present disclosure;
fig. 4 shows a block diagram of a three-dimensional reconstruction apparatus of a three-dimensional space according to an embodiment of the present disclosure;
fig. 5 shows a schematic structural diagram of a three-dimensional reconstruction apparatus of a three-dimensional space according to an embodiment of the present disclosure;
FIG. 6 shows a schematic diagram of a storage medium according to an embodiment of the disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described below in detail and completely with reference to the accompanying drawings of the embodiments of the present disclosure. It is to be understood that the described embodiments are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the disclosure without any inventive step, are within the scope of protection of the disclosure.
Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure belongs. The use of "first," "second," and similar terms in this disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly. To maintain the following description of the embodiments of the present disclosure clear and concise, a detailed description of some known functions and components have been omitted from the present disclosure.
Flow charts are used in this disclosure to illustrate steps of methods according to embodiments of the disclosure. It should be understood that the preceding and following steps are not necessarily performed in the exact order in which they are performed. Rather, various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or steps may be removed from the processes.
With the popularization of panoramic cameras, panoramic images are becoming more common. In recent years, along with the hot of artificial intelligence technology and the strong demand of vast people for renting and purchasing rooms, more and more room renting and selling platforms use panoramic cameras to acquire panoramic images of room sources. In order to enhance the experience of network house-viewing, it is desirable to be able to effectively use panoramic images of these house sources for three-dimensional reconstruction to produce a more realistic representation of a three-dimensional room.
Motion recovery Structure (SfM) is a technique for calculating the pose (position and attitude) of a camera and the three-dimensional Structure of an object according to the computer vision theory and performing three-dimensional reconstruction. The SfM-based three-dimensional reconstruction technology generally needs to take multiple panoramic images by using a camera, and performs three-dimensional reconstruction on a three-dimensional space by acquiring a translation vector and a rotation matrix of the camera and position information of pixel points or geometric structures in the panoramic images. However, SfM-based three-dimensional reconstruction methods for building interior spaces remain a challenging problem. The difficulty of this problem mainly derives from the following two aspects: firstly, the internal space of a building often has a large number of texture missing areas such as white walls and the like, and feature points are difficult to extract in the area, so that feature matching is difficult to form for subsequent three-dimensional reconstruction; secondly, a large amount of disordered and shielded objects formed by space structures exist in indoor scenes such as internal spaces of buildings, and extraction and detection of three-dimensional reconstruction key information such as feature points and wall corners are not facilitated.
At least one embodiment of the present disclosure provides a three-dimensional reconstruction method of a three-dimensional space, a three-dimensional reconstruction apparatus, and a computer-readable storage medium. Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.
A method of three-dimensional reconstruction of a three-dimensional space according to an embodiment of the present disclosure is described below with reference to fig. 1, which fig. 1 shows a flow chart of the method 100 of three-dimensional reconstruction of a three-dimensional space. The three-dimensional reconstruction method 100 includes steps S101 to S103 as follows:
step S101: respectively carrying out geometric shape detection and geometric relation detection on a plurality of panoramic images of the three-dimensional space by utilizing the geometric prior of the three-dimensional space, and determining a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction;
step S102: according to the geometric prior of the three-dimensional space, constructing geometric constraints aiming at the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection, and determining the translation vector and the position information of a camera for shooting the panoramic image for three-dimensional reconstruction according to the constructed geometric constraints;
step S103: and performing three-dimensional reconstruction on the three-dimensional space by using the rotation matrix, the translation vector and the position information.
Alternatively, the three-dimensional reconstruction method shown in fig. 1 may be applied to an indoor space of a building, for example, three-dimensional reconstruction may be performed on three-dimensional rooms, so as to obtain a three-dimensional model of each room. For example, the indoor space of the building herein may include a bottom surface and a top surface parallel to each other, and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form an at least partial enclosure. Alternatively, the indoor space may include a residence, an office, etc., and may include one or more subspaces, e.g., the plurality of subspaces being located in the same floor. For example, when the indoor space is a house, the indoor space may include a living room subspace, a main lying subspace, a sub-lying subspace, a balcony subspace, a toilet subspace, a kitchen subspace, and the like. The above description of the indoor space is merely an example, and the present disclosure includes but is not limited thereto. The method for three-dimensional reconstruction of the three-dimensional space can be to perform three-dimensional reconstruction after splicing a plurality of two-dimensional images shot by a common camera, or can be to perform three-dimensional reconstruction of panoramic images shot by a panoramic camera. In the disclosed embodiments, a plurality of panoramic images may be taken for a three-dimensional space by a panoramic camera, thereby performing a fast and efficient three-dimensional reconstruction process using the panoramic images.
As shown in fig. 1, in step S101, using a geometric prior of a three-dimensional space, a plurality of panoramic images of the three-dimensional space are subjected to geometric shape detection and geometric relationship detection, respectively, and a rotation matrix of a camera for taking the panoramic images for three-dimensional reconstruction is determined.
In the disclosed embodiment, the geometric prior of the three-dimensional space may be a geometric prior for a particular three-dimensional space. For example, for a three-dimensional space that is an indoor space of a building, at least one of the following basic geometrical relationships may be satisfied based on the basic geometry of the room: aiming at the co-linear relationship that the same straight line is superposed with each other under different visual angles, the coplanar relationship, the parallel relationship and the vertical relationship between the straight line and the straight line, the ceiling and the ground in a room are parallel to each other, the opposite wall surfaces are parallel to each other, the adjacent wall surfaces are perpendicular to each other, the wall surfaces are perpendicular to the ceiling or the ground, or three surfaces intersected at one point are perpendicular to each other in pairs, and the like. According to the examples of the geometric prior, more specific geometric constraints can be constructed for a plurality of panoramic images acquired in a three-dimensional space, so that each parameter for three-dimensional reconstruction is further determined in an auxiliary manner, and the accuracy of a three-dimensional reconstruction result is improved.
In the embodiment of the present disclosure, considering that lines in a panoramic image of a three-dimensional space generally generate a relatively strong distortion, the panoramic image may be first corrected into a perspective image, and a subsequent image processing process may be performed based on the corrected perspective image. In one example, the panoramic image may be projected as a cubic perspective image by cubic projection; in another example, the panoramic image may be projected as a latitude and longitude perspective image by latitude and longitude projection; in still another example, the panoramic image may also be projected as a cylindrical perspective image or the like by cylindrical projection. The above-mentioned correction method for the panoramic image is only an example, and in practical application, any required correction method for the panoramic image may be adopted, which is not limited herein. For example, for a panoramic image of a three-dimensional house type, a cubic projection is selected to project the panoramic image into a cubic perspective image with six viewing angles, namely front and back, up and down, left and right, and three manhattan main directions (such as three coordinate axis directions of x, y and z) orthogonal to each other are determined according to the cubic perspective image.
After the panoramic image is corrected into a perspective image, optionally, the corrected perspective image may be subjected to geometry detection and geometry relationship detection using the aforementioned geometry prior of the three-dimensional space. For example, the corrected perspective image may be detected by using some geometric shape or geometric relationship detection tools, or the geometric shape or geometric relationship may be detected by using a trained neural network in a deep learning manner. Alternatively, the geometry detection of the perspective image may include Line detection of the perspective image, for example, a Line Segment detection tool such as a Line Segment Detector may be called to perform Line detection of the perspective image. Of course, the geometry detection of the perspective image may also include other ways, for example, some special shapes, such as a circle, a square, etc., may be detected according to a specific application scenario, which is not limited herein.
After geometry detection including line detection of the fluoroscopic images, optionally, at least one vanishing point in three-dimensional space may be determined from the detected lines. According to the embodiment of the present disclosure, when the three-dimensional space is an indoor space of a building, i.e., a room, in consideration of the geometric prior of the three-dimensional space, the adjacent walls, ceilings, and floors in the room may be considered to be perpendicular to each other, so that vanishing points in three directions orthogonal to each other (i.e., along three manhattan main directions) may be generated. After detecting a straight line from the perspective image, vanishing points for each direction can be determined therefrom. In one example, vanishing points in various directions orthogonal to each other may be voted according to the detected straight line to obtain an accurate position of the vanishing point. For example, all possible positions of the corresponding vanishing point may be scored according to the direction in which the detected straight line is directed, and the highest-scoring result may be selected as the position of the corresponding vanishing point determined by the straight line. In another example, the vanishing point location may also be detected using a trained neural network for vanishing point detection to obtain an accurate location of the vanishing point. Optionally, the determined vanishing point may be connected to a midpoint of the detected straight line to determine whether a connecting line between the vanishing point and the midpoint of the straight line and the straight line satisfy a collinear relationship within a certain threshold, and the specific position of the vanishing point is continuously adjusted in an iterative manner.
After determining the vanishing point in the three-dimensional space, according to an embodiment of the present disclosure, the determined vanishing point may also be utilized to further refine the straight line detection result in the three-dimensional space. For example, when the three-dimensional space is a room inside a building, the detected straight line may be optimized based on three manhattan principal directions perpendicular to each other in the room. Optionally, the result of the line detection may be optimized by at least one of: the vanishing point and the straight line in a certain distance range or angle range around the connecting line of a certain straight line can be mutually combined; the straight line in a certain direction can be supplemented according to vanishing points in the three directions aiming at the missing of the straight line in the direction (for example, in three manhattan main directions xyz, straight lines in two directions xy exist, and the straight line which cannot be detected in the direction z can be supplemented according to the straight lines in the two directions xy); and certain straight lines which do not meet the relationship can be filtered according to the relationship of the intersection ratio of the straight lines, so that a more accurate straight line detection result is obtained.
The above describes a process of performing geometry detection on a panoramic image of a three-dimensional space using a geometric prior of the three-dimensional space. After obtaining the geometry detection result (e.g., straight line detection result) for the fluoroscopic image, it is also possible to perform geometry relation detection based on at least the result of the straight line detection. In the process of detecting the geometric relationship, the detected group of straight lines satisfying a certain geometric relationship may be a pair of straight lines formed by two or a set of straight lines including more than two straight lines. When geometric constraints are constructed subsequently by using geometric relationships, optionally, constraints may be constructed by directly using the geometric relationships between the above straight line pairs, geometric constraints may also be constructed by using pairwise combinations of a plurality of straight lines in the straight line set, geometric constraints may also be constructed by directly using a plurality of straight lines, and the like, which is not limited herein. In the specific example that follows, the geometric constraints are built primarily for pairs of lines that satisfy the geometric relationship. Further, according to embodiments of the present disclosure, the geometric relationship detection may include at least one of: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection. Alternatively, when the three-dimensional space is a room in a building, for the detection of the coplanar straight line groups in the geometric relationship detection, a plane in which the coplanar straight line groups are commonly located may be defined to conform to one of the three manhattan principal directions xyz. In addition, the detected relative relationship between the coplanar straight lines can be further classified into types for the construction of geometric constraints for specific geometric relationships later, for example, the coplanar straight line group can be further classified into: a parallel coplanar straight line group (e.g., two coplanar straight lines along the x direction), a perpendicular coplanar straight line group (e.g., two coplanar straight lines along the x and y directions, respectively), a collinear coplanar straight line group (e.g., two coplanar straight lines along the x direction and with close end points), and so on.
According to at least one embodiment of the present disclosure, the specific manner of performing the geometric shape detection and the geometric relationship detection on the panoramic image of the three-dimensional space by using the geometric prior of the three-dimensional space may be respectively applied to each panoramic image of the plurality of panoramic images, or may be applied to a part of the panoramic images of the plurality of panoramic images, and the specific application manner is not limited. After the above-described geometry detection and geometry relation detection are completed, the detection result may also be used to determine a rotation matrix of a camera for taking a panoramic image for three-dimensional reconstruction. Alternatively, after at least straight line detection results of the at least two perspective images have been acquired and vanishing points are determined, the rotation matrices of the cameras corresponding to the at least two perspective images may be determined accordingly. Or, alternatively, when a camera for photographing a panoramic image is equipped with an Inertial Measurement Unit (IMU), the rotational posture of the camera when photographing the panoramic image may be measured by the IMU to obtain an initial rotation matrix, and data recording the initial rotation matrix may be included in the panoramic image data, for example, the initial rotation matrices of the cameras that photograph different panoramic images are different; subsequently, the initial rotation matrix may be adjusted using at least the line detection results and vanishing points of the acquired at least one fluoroscopic image. For example, in a room inside a building, the rotation matrix of the camera corresponding to the at least one perspective image may be further optimized for the three manhattan principal directions by minimizing the distance of the detected straight line in the same direction from the vanishing point.
After the rotation matrix of one or more cameras is acquired, the rotation matrix of the camera can be applied to perform subsequent three-dimensional reconstruction operation, and a bundle adjustment process after three-dimensional reconstruction can also be introduced into the rotation matrix of the camera, that is, reprojection correction is performed.
In step S102, geometric constraints are constructed for the geometric shapes and geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection according to geometric prior of the three-dimensional space, and translation vectors and position information of cameras for shooting panoramic images for three-dimensional reconstruction are determined according to the constructed geometric constraints.
According to the embodiment of the disclosure, feature point detection can be performed on a plurality of perspective images respectively, and feature point matching can be performed according to the result of the feature point detection. Optionally, as a result of performing feature point matching on the plurality of perspective images, the same straight line respectively existing on the at least two perspective images may be obtained, or at least two straight lines respectively existing on the at least two perspective images and satisfying the previously detected geometric relationship may be obtained, so as to construct a corresponding geometric constraint.
Alternatively, the two different types of light beams may be directed to the presence of at least two different types of light beams, respectivelyConstructing a collinearity constraint according to the same straight line on the image, and determining a translation vector and position information of a camera for shooting the panoramic image for three-dimensional reconstruction according to at least the constructed collinearity constraint. For example, when for a panoramic image acquired by camera i, the midpoint of the detected straight line l
Figure BDA0002325004900000131
Has a depth of
Figure BDA0002325004900000132
Can be combined with
Figure BDA0002325004900000133
Expressed as formula (1):
Figure BDA0002325004900000134
wherein T isiIs the translation vector of the camera i and,
Figure BDA0002325004900000135
is the optical center of the camera i
Figure BDA0002325004900000136
The unit vector of the ray of (2).
Similarly, when the straight line l is located in the panoramic image acquired by the camera j at the same time, the midpoint P of the straight line l detected on the panoramic image acquired by the camera j may bel jExpressed as formula (2):
Figure BDA0002325004900000137
wherein, TjIs the translation vector of the camera j,
Figure BDA0002325004900000138
is the optical center of camera j to Pl jThe unit vector of the ray of (a),
Figure BDA0002325004900000139
is Pl jOf the depth of (c).
On this basis, a collinearity constraint can be constructed for the straight line l represented in the panoramic image acquired by the camera i and the straight line l represented in the panoramic image acquired by the camera j, and the geometric constraint is expressed as formula (3):
Figure BDA00023250049000001310
wherein N isl 1、Nl 2A unit vector in two orthogonal directions being a straight line l (e.g. for a room in a building room, there may be any two of the three manhattan principal directions xyz, e.g. xy, xz or yz). Accordingly, translation vectors T of camera i and camera j can be respectively determined according to constructed collinearity constraintsi、TjAnd position information of the straight line l (which may be a midpoint, for example)
Figure BDA00023250049000001318
Pl jOr depth of
Figure BDA00023250049000001311
)。
Furthermore, optionally, geometric constraints for the corresponding geometric relationships may also be constructed for at least two straight lines that respectively exist on the at least two fluoroscopic images and satisfy the previously detected geometric relationships, so as to determine translation vectors and position information of the camera that captures the panoramic image for three-dimensional reconstruction, at least according to the constructed geometric constraints.
For example, when for a panoramic image acquired by camera j, the midpoint of the detected straight line m
Figure BDA00023250049000001312
Has a depth of
Figure BDA00023250049000001313
Can be combined with
Figure BDA00023250049000001314
Expressed as formula (4):
Figure BDA00023250049000001315
wherein, TjIs the translation vector of the camera j,
Figure BDA00023250049000001316
is the optical center of the camera j to
Figure BDA00023250049000001317
The unit vector of the ray of (2).
On this basis, when the straight line l and the straight line m satisfy the coplanar geometric relationship, a coplanarity constraint may be constructed for the straight line l represented in the panoramic image acquired by the camera i and the straight line m represented in the panoramic image acquired by the camera j, and the geometric constraint is expressed as formula (5):
Figure BDA0002325004900000141
wherein N isl,mIs the normal vector of the plane in which the pair of lines lie. For example, for a room in a building room, for three manhattan principal directions xyz, when the plane of the line l and the line m is the xy plane, Nl,mMay point in the z direction. Accordingly, translation vectors T of camera i and camera j can be respectively determined according to constructed collinearity constraintsi、TjAnd position information of the straight lines l, m (which may be, for example, midpoints)
Figure BDA0002325004900000142
Or depth of
Figure BDA0002325004900000143
Figure BDA0002325004900000144
)。
In step S103, three-dimensional reconstruction of a three-dimensional space may be performed using the rotation matrix, the translation vector, and the position information.
According to the embodiments of the present disclosure, after the rotation matrix, the translation vector, and the position information required for three-dimensional reconstruction of the camera for photographing the panoramic image have been acquired, three-dimensional reconstruction of a three-dimensional space may be performed using, for example, SfM. In one example, the camera pose can be calculated for the initial image pair by using an incremental SfM method, and continuous iterative optimization is performed to obtain a three-dimensional reconstruction result; in another example, the global SfM approach can also be used to spread the computation error over the individual rotations or translations for all camera poses to obtain the three-dimensional reconstruction results.
Optionally, after the result of the three-dimensional reconstruction is obtained, in order to further improve the accuracy of the three-dimensional reconstruction, bundle adjustment (bundleadjust) may be performed on the result of the three-dimensional reconstruction of the three-dimensional space according to the geometric prior of the three-dimensional space. In particular, in one example, at least one straight line obtained in a three-dimensional reconstruction of a three-dimensional space may be parameterized and bundle-adjusted with a collinearity constraint for the at least one straight line. For example, for a straight line l, with AlRepresents the direction of the straight line l and calculates the perpendicular Λ of the straight line using a specific algorithm (e.g., Levenberg-Marquardt algorithm can be used with a Ceres library invoked for non-linear optimization)lAnd the collinearity constraint is considered so that the formula (6) is satisfied to perform the bundle adjustment.
Al·Λl＝0 (6)
In another example, at least two coplanar straight lines obtained in a three-dimensional reconstruction of a three-dimensional space may be parameterized and bundle-adjusted with a coplanarity constraint for the at least two coplanar straight lines. For example, for two coplanar straight lines l and m, they are respectively parameterized as ΛlAnd ΛmAnd using a specific algorithm (e.g., Levenberg-Marquardt algorithm, which may be called for by Ceres library) to calculate the distance A between lines normal to the plane where l and m are commonlmConsider the coplanarity constraint such that equation (7) is satisfied for assemblyBeam adjustment:
(Λl-Λm)·Alm＝0 (7)
the cluster adjustment mode can optimize the reprojection error of the three-dimensional reconstruction method by combining corresponding constraint, and obtain a more accurate three-dimensional reconstruction result. In the specific optimization process, the bundle adjustment mode using the geometric constraint may be considered, and other bundle adjustment modes may also be considered to optimize the result of the three-dimensional reconstruction. Optionally, corresponding weights may be given to different bundling adjustment manners to adapt to different specific application scenarios.
Fig. 2 shows an example of a three-dimensional house view (or referred to as a stereo house view) constructed by a three-dimensional reconstruction method according to an embodiment of the present disclosure for a panoramic image of each room in a house in one example. As can be seen from fig. 2, the exemplary three-dimensional house type diagram shows various subspaces (e.g., a living room subspace, a main lying subspace, a sub-lying subspace, a balcony subspace, a toilet subspace, a kitchen subspace, etc.) of a house, a position arrangement of the various subspaces, and the like.
According to the three-dimensional reconstruction method provided by the embodiment of the disclosure, the specific geometric prior of the three-dimensional space, particularly the internal space of a building, can be utilized to perform geometric shape detection and geometric relation detection on a panoramic image, and the panoramic image is used as a constraint condition to obtain various parameters required by three-dimensional reconstruction, so that the process of realizing the three-dimensional reconstruction is assisted. According to the three-dimensional reconstruction method disclosed by the invention, the problems caused by texture loss of the internal space of the building or shielding of furniture interior trim and the like can be overcome, so that the three-dimensional reconstruction is more accurately and efficiently carried out to obtain the accurate three-dimensional internal structure of the internal space of the building.
According to the internal space structure of the room, for example, constructed by the three-dimensional reconstruction method, a three-dimensional room model with a more realistic sense can be provided for users of network house watching, and the three-dimensional room model is displayed more vividly aiming at the house type structure, the indoor space relationship, the internal size of the room and the like of each house source, so that the immersive house watching experience can be obtained by remote network users.
Next, a three-dimensional reconstruction apparatus of a three-dimensional space according to an embodiment of the present disclosure is described with reference to fig. 3. Fig. 3 shows a block diagram of a three-dimensional reconstruction apparatus 300 according to an embodiment of the present disclosure. As shown in fig. 3, the three-dimensional reconstruction apparatus 300 includes a detection unit 310, a determination unit 320, and a reconstruction unit 330. The three-dimensional reconstruction apparatus 300 may include other components in addition to these units, however, since these components are not related to the contents of the embodiments of the present disclosure, illustration and description thereof are omitted herein. Furthermore, since the specific details of the following operations performed by the three-dimensional reconstruction apparatus 300 according to the embodiment of the present disclosure are the same as those described above with reference to fig. 1 to 2, a repeated description of the same details is omitted herein to avoid repetition.
The three-dimensional reconstruction apparatus 300 in fig. 3 may be applied to an indoor space of a building, for example, three-dimensional reconstruction may be performed on three-dimensional rooms, so as to obtain a three-dimensional model of each room. Specifically, the building indoor space herein may include a bottom surface and a top surface parallel to each other, and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form at least a partial enclosure. Alternatively, the indoor space may include a residence, an office, etc., and may include one or more subspaces, e.g., the plurality of subspaces being located in the same floor. For example, when the indoor space is a house, the indoor space may include a living room subspace, a main lying subspace, a sub-lying subspace, a balcony subspace, a toilet subspace, a kitchen subspace, and the like. The above description of the indoor space is merely an example, and the present disclosure includes but is not limited thereto. The device for three-dimensional reconstruction of three-dimensional space can be used for three-dimensional reconstruction after splicing a plurality of two-dimensional images shot by a common camera, and can also be used for three-dimensional reconstruction of panoramic images shot by a panoramic camera. In the disclosed embodiments, a plurality of panoramic images may be taken for a three-dimensional space by a panoramic camera, thereby performing a fast and efficient three-dimensional reconstruction process using the panoramic images.
As shown in fig. 3, the detection unit 310 performs geometry detection and geometry relationship detection on a plurality of panoramic images of a three-dimensional space, respectively, using a geometric prior of the three-dimensional space, and determines a rotation matrix of a camera for taking the panoramic images for three-dimensional reconstruction.
In at least one embodiment of the present disclosure, a geometric prior of a three-dimensional space may be a geometric prior for a particular three-dimensional space. For example, for a three-dimensional space that is an indoor space of a building, at least one of the following basic geometrical relationships may be satisfied based on the basic geometry of the room: aiming at the co-linear relationship that the same straight line is superposed with each other under different visual angles, the coplanar relationship, the parallel relationship and the vertical relationship between the straight line and the straight line, the ceiling and the ground in a room are parallel to each other, the opposite wall surfaces are parallel to each other, the adjacent wall surfaces are perpendicular to each other, the wall surfaces are perpendicular to the ceiling or the ground, or three surfaces intersected at one point are perpendicular to each other in pairs, and the like. According to the examples of the geometric prior, more specific geometric constraints can be constructed for a plurality of panoramic images acquired in a three-dimensional space, so that each parameter for three-dimensional reconstruction is further determined in an auxiliary manner, and the accuracy of a three-dimensional reconstruction result is improved. The plurality of panoramic images for three-dimensional reconstruction of the three-dimensional space may be stored in a local memory, for example, or may be acquired through a network (server or cloud, etc.).
In at least one embodiment of the present disclosure, considering that lines in a panoramic image of a three-dimensional space generally have a relatively strong distortion, the detection unit 310 may first correct the panoramic image into a perspective image and perform a subsequent image processing process based on the corrected perspective image. In one example, the panoramic image may be projected as a cubic perspective image by cubic projection; in another example, the panoramic image may be projected as a latitude and longitude perspective image by latitude and longitude projection; in still another example, the panoramic image may also be projected as a cylindrical perspective image or the like by cylindrical projection. The above-mentioned correction method for the panoramic image is only an example, and in practical application, any required correction method for the panoramic image may be adopted, which is not limited herein. For example, for a panoramic image of a three-dimensional house type, a cubic projection is selected to project the panoramic image into a cubic perspective image with six viewing angles, namely front and back, up and down, left and right, and three manhattan main directions (such as three coordinate axis directions of x, y and z) orthogonal to each other are determined according to the cubic perspective image.
After the panoramic image is corrected to a perspective image, the detection unit 310 may optionally perform geometry detection and geometric relationship detection on the corrected perspective image by using the aforementioned geometric prior of the three-dimensional space. For example, the corrected perspective image may be detected by using some geometric shape or geometric relationship detection tools, or the geometric shape or geometric relationship may be detected by using a trained neural network in a deep learning manner. Alternatively, the geometry detection of the perspective image may include Line detection of the perspective image, for example, a Line Segment detection tool such as a Line Segment Detector may be called to perform Line detection of the perspective image. Of course, the geometry detection of the perspective image may also include other ways, for example, some special shapes, such as a circle, a square, etc., may be detected according to a specific application scenario, which is not limited herein.
After performing geometry detection including line detection on the fluoroscopic image, the detecting unit 310 may optionally determine at least one vanishing point in the three-dimensional space according to the detected line. According to the embodiment of the present disclosure, when the three-dimensional space is an indoor space of a building, i.e., a room, in consideration of the geometric prior of the three-dimensional space, the adjacent walls, ceilings, and floors in the room may be considered to be perpendicular to each other, so that vanishing points in three directions orthogonal to each other (i.e., along three manhattan main directions) may be generated. After detecting a straight line from the perspective image, vanishing points for each direction can be determined therefrom. In one example, vanishing points in various directions orthogonal to each other may be voted according to the detected straight line to obtain an accurate position of the vanishing point. For example, all possible positions of the corresponding vanishing point may be scored according to the direction in which the detected straight line is directed, and the highest-scoring result may be selected as the position of the corresponding vanishing point determined by the straight line. In another example, the vanishing point location may also be detected using a trained neural network for vanishing point detection to obtain an accurate location of the vanishing point. Optionally, the determined vanishing point may be connected to a midpoint of the detected straight line to determine whether a connecting line between the vanishing point and the midpoint of the straight line and the straight line satisfy a collinear relationship within a certain threshold, and the specific position of the vanishing point is continuously adjusted in an iterative manner.
After determining the vanishing point in the three-dimensional space, according to the embodiment of the present disclosure, the detecting unit 310 may further refine the straight line detection result in the three-dimensional space by using the determined vanishing point. For example, when the three-dimensional space is a room inside a building, the detected straight line may be optimized based on three manhattan principal directions perpendicular to each other in the room. Optionally, the result of the line detection may be optimized by at least one of: the vanishing point and the straight line in a certain distance range or angle range around the connecting line of a certain straight line can be mutually combined; the straight line in a certain direction can be supplemented according to vanishing points in the three directions aiming at the missing of the straight line in the direction (for example, in three manhattan main directions xyz, straight lines in two directions xy exist, and the straight line which cannot be detected in the direction z can be supplemented according to the straight lines in the two directions xy); and certain straight lines which do not meet the relationship can be filtered according to the relationship of the intersection ratio of the straight lines, so that a more accurate straight line detection result is obtained.
The above describes the process of the detection unit 310 performing the geometry detection on the panoramic image of the three-dimensional space by using the geometry prior of the three-dimensional space. After obtaining the geometry detection result (e.g., straight line detection result) for the fluoroscopic image, the detection unit 310 may also perform geometry relation detection based on at least the result of the straight line detection. In the process of detecting the geometric relationship, the detected group of straight lines satisfying a certain geometric relationship may be a pair of straight lines formed by two or a set of straight lines including more than two straight lines. When geometric constraints are constructed subsequently by using geometric relationships, optionally, constraints may be constructed by directly using the geometric relationships between the above straight line pairs, geometric constraints may also be constructed by using pairwise combinations of a plurality of straight lines in the straight line set, geometric constraints may also be constructed by directly using a plurality of straight lines, and the like, which is not limited herein. In the specific example that follows, the geometric constraints are built primarily for pairs of lines that satisfy the geometric relationship. Further, according to embodiments of the present disclosure, the geometric relationship detection may include at least one of: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection. Alternatively, when the three-dimensional space is a room in a building, for the detection of the coplanar straight line groups in the geometric relationship detection, a plane in which the coplanar straight line groups are commonly located may be defined to conform to one of the three manhattan principal directions xyz. In addition, the detected relative relationship between the coplanar straight lines can be further classified into types for the construction of geometric constraints for specific geometric relationships later, for example, the coplanar straight line group can be further classified into: a parallel coplanar straight line group (e.g., two coplanar straight lines along the x direction), a perpendicular coplanar straight line group (e.g., two coplanar straight lines along the x and y directions, respectively), a collinear coplanar straight line group (e.g., two coplanar straight lines along the x direction and with close end points), and so on.
According to at least one embodiment of the present disclosure, the specific manner of performing the geometric shape detection and the geometric relationship detection on the panoramic image of the three-dimensional space by using the geometric prior of the three-dimensional space may be respectively applied to each panoramic image of the plurality of panoramic images, or may be applied to a part of the panoramic images of the plurality of panoramic images, and the specific application manner is not limited. After completing the above-described geometry detection and geometry relationship detection, the detection unit 310 may also determine a rotation matrix of a camera that captures a panoramic image for three-dimensional reconstruction using the detection result. Alternatively, after at least straight line detection results of the at least two perspective images have been acquired and vanishing points are determined, the rotation matrices of the cameras corresponding to the at least two perspective images may be determined accordingly. Or, alternatively, when a camera for photographing a panoramic image is equipped with an Inertial Measurement Unit (IMU), the rotational posture of the camera when photographing the panoramic image may be measured by the IMU to obtain an initial rotation matrix, and data recording the initial rotation matrix may be included in the panoramic image data, for example, the initial rotation matrix of the camera photographing different panoramic images is different; subsequently, the initial rotation matrix may be adjusted using at least the line detection results and vanishing points of the acquired at least one fluoroscopic image. For example, in a room inside a building, the rotation matrix of the camera corresponding to the at least one perspective image may be further optimized for the three manhattan principal directions by minimizing the distance of the detected straight line in the same direction from the vanishing point.
After the rotation matrix of one or more cameras is acquired, the rotation matrix of the camera can be applied to perform subsequent three-dimensional reconstruction operation, and a bundle adjustment process after three-dimensional reconstruction can also be introduced into the rotation matrix of the camera, that is, reprojection correction is performed.
The determination unit 320 constructs geometric constraints for the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection according to the geometric prior of the three-dimensional space, and determines the translation vector and the position information of the camera for photographing the panoramic image for three-dimensional reconstruction according to the constructed geometric constraints.
According to the embodiment of the present disclosure, the determining unit 320 may perform feature point detection on the plurality of perspective images, respectively, and perform feature point matching according to a result of the feature point detection. Optionally, as a result of performing feature point matching on the plurality of perspective images, the same straight line respectively existing on the at least two perspective images may be obtained, or at least two straight lines respectively existing on the at least two perspective images and satisfying the previously detected geometric relationship may be obtained, so as to construct a corresponding geometric constraint.
Alternatively, the determination unit 320 may construct a collinearity constraint for the same straight line respectively existing on the at least two fluoroscopic images, and determine the translation vector and the position information of the camera taking the panoramic image for the three-dimensional reconstruction from at least the constructed collinearity constraint. For example, when for a panoramic image acquired by camera i, the midpoint of the detected straight line l
Figure BDA0002325004900000191
Has a depth of
Figure BDA0002325004900000192
Can be combined with
Figure BDA0002325004900000193
Expressed as formula (1), wherein TiIs the translation vector of the camera i and,
Figure BDA0002325004900000194
is the optical center of the camera i
Figure BDA0002325004900000195
The unit vector of the ray of (2).
Similarly, when the straight line l is located in the panoramic image acquired by the camera j at the same time, the midpoint P of the straight line l detected on the panoramic image acquired by the camera j may bel jExpressed as formula (2), wherein TjIs the translation vector of the camera j,
Figure BDA0002325004900000196
is the optical center of camera j to Pl jThe unit vector of the ray of (a),
Figure BDA0002325004900000197
is Pl jOf the depth of (c).
On this basis, a collinearity constraint can be constructed for the straight line l represented in the panoramic image acquired by the camera i and the straight line l represented in the panoramic image acquired by the camera j, and the geometric constraint is expressed as formula (3). Wherein N isl 1、Nl 2A unit vector in two orthogonal directions being a straight line l (e.g. for a room in a building room, there may be any two of the three manhattan principal directions xyz, e.g. xy, xz or yz). Accordingly, translation vectors T of camera i and camera j can be respectively determined according to constructed collinearity constraintsi、TjAnd position information of the straight line l (which may be a midpoint, for example)
Figure BDA0002325004900000201
Pl jOr depth of
Figure BDA0002325004900000202
)。
Furthermore, optionally, geometric constraints for the corresponding geometric relationships may also be constructed for at least two straight lines that respectively exist on the at least two fluoroscopic images and satisfy the previously detected geometric relationships, so as to determine translation vectors and position information of the camera that captures the panoramic image for three-dimensional reconstruction, at least according to the constructed geometric constraints.
For example, when for a panoramic image acquired by camera j, the midpoint of the detected straight line m
Figure BDA0002325004900000203
Has a depth of
Figure BDA0002325004900000204
Can be combined with
Figure BDA0002325004900000205
Expressed as formula (4). Wherein, TjIs the translation vector of the camera j,
Figure BDA0002325004900000206
is the optical center of the camera j to
Figure BDA0002325004900000207
The unit vector of the ray of (2).
On the basis, when the straight line l and the straight line m satisfy the coplanar geometric relationship, a coplanar constraint can be constructed for the straight line l represented in the panoramic image acquired by the camera i and the straight line m represented in the panoramic image acquired by the camera j, and the geometric constraint is expressed as formula (5), wherein N isl,mIs the normal vector of the plane in which the pair of lines lie. For example, for a room in a building room, for three manhattan principal directions xyz, when the plane of the line l and the line m is the xy plane, Nl,mMay point in the z direction. Accordingly, the planes of camera i and camera j can be determined separately from the constructed collinearity constraintsShift vector Ti、TjAnd position information of the straight lines l, m (which may be, for example, midpoints)
Figure BDA0002325004900000208
Or depth of
Figure BDA0002325004900000209
)。
The reconstruction unit 330 may perform three-dimensional reconstruction for a three-dimensional space using the rotation matrix, the translation vector, and the position information.
According to the embodiment of the present disclosure, after the rotation matrix, the translation vector, and the position information required for three-dimensional reconstruction of the camera for photographing the panoramic image have been acquired, the reconstruction unit 330 may three-dimensionally reconstruct the three-dimensional space using, for example, SfM. In one example, the camera pose can be calculated for the initial image pair by using an incremental SfM method, and continuous iterative optimization is performed to obtain a three-dimensional reconstruction result; in another example, the global SfM approach can also be used to spread the computation error over the individual rotations or translations for all camera poses to obtain the three-dimensional reconstruction results.
Optionally, after obtaining the three-dimensional reconstruction result, in order to further improve the accuracy of the three-dimensional reconstruction, the reconstruction unit 330 may further perform Bundle Adjustment (Bundle Adjustment) on the three-dimensional reconstruction result of the three-dimensional space according to the geometric prior of the three-dimensional space. In particular, in one example, at least one straight line obtained in a three-dimensional reconstruction of a three-dimensional space may be parameterized and bundle-adjusted with a collinearity constraint for the at least one straight line. For example, for a straight line l, with AlRepresents the direction of the straight line l and calculates the perpendicular Λ of the straight line using a specific algorithm (e.g., a Ceres library can be invoked using the Levenberg-Marquardt algorithm)lAnd the collinearity constraint is considered so that the formula (6) is satisfied to perform the bundle adjustment.
In another example, at least two coplanar straight lines obtained in a three-dimensional reconstruction of a three-dimensional space may be parameterized and utilized with a common for the at least two coplanar straight linesAnd (5) performing clustering adjustment by surface constraint. For example, for two coplanar straight lines l and m, they are respectively parameterized as ΛlAnd ΛmAnd using a specific algorithm (e.g., Levenberg-Marquardt algorithm, which may be called for by Ceres library) to calculate the distance A between lines normal to the plane where l and m are commonlmThe coplanarity constraint is considered so that equation (7) is satisfied to perform bundle adjustment.
The cluster adjustment mode can optimize the reprojection error of the three-dimensional reconstruction method by combining corresponding constraint, and obtain a more accurate three-dimensional reconstruction result. In the specific optimization process, the bundle adjustment mode using the geometric constraint may be considered, and other bundle adjustment modes may also be considered to optimize the result of the three-dimensional reconstruction. Optionally, corresponding weights may be given to different bundling adjustment manners to adapt to different specific application scenarios.
Fig. 2 shows an example of a three-dimensional house view (or referred to as a stereo house view) constructed by a three-dimensional reconstruction method according to an embodiment of the present disclosure for a panoramic image of each room in a house in one example. As can be seen from fig. 2, the exemplary three-dimensional house type diagram shows various subspaces (e.g., a living room subspace, a main lying subspace, a sub-lying subspace, a balcony subspace, a toilet subspace, a kitchen subspace, etc.) of a house, a position arrangement of the various subspaces, and the like.
According to the three-dimensional reconstruction device provided by the embodiment of the disclosure, the specific geometric prior of a three-dimensional space, particularly an internal space of a building, can be utilized to perform geometric shape detection and geometric relation detection on a panoramic image, and the panoramic image is used as a constraint condition to obtain various parameters required by three-dimensional reconstruction, so that the process of realizing the three-dimensional reconstruction is assisted. According to the three-dimensional reconstruction device disclosed by the invention, the problems caused by texture loss of the inner space of the building or shielding of furniture interior trim and the like can be overcome, so that the three-dimensional reconstruction is more accurately and efficiently carried out, and the accurate three-dimensional inner structure of the inner space of the building is obtained.
According to the internal space structure of the room, for example, constructed by the three-dimensional reconstruction device disclosed by the embodiment of the disclosure, a three-dimensional room model with a more realistic sense can be provided for a user of network room watching, and the three-dimensional room model is more vividly displayed aiming at the house type structure, the indoor space relationship, the internal size of the room and the like of each house source, so that the immersive room watching experience can be obtained by a remote network user.
At least one embodiment of the present disclosure further provides a three-dimensional reconstruction device for a three-dimensional space. Fig. 4 is a schematic diagram of a three-dimensional reconstruction apparatus 400 according to an embodiment of the present disclosure.
For example, as shown in fig. 4, a three-dimensional reconstruction apparatus 400 may include one or more memories 410 and one or more processors 420. It should be noted that the components of the three-dimensional reconstruction apparatus 400 shown in fig. 4 are only exemplary and not limiting, and the three-dimensional reconstruction apparatus 400 may have other components according to the actual application.
For example, memory 410 is used for non-transitory storage of computer program instructions; the processor 420 is configured to execute computer program instructions, which when executed by the processor 420, perform one or more steps of a method for three-dimensional reconstruction of a three-dimensional space according to any of the embodiments described above.
For example, the computer program instructions, when executed by the processor 420, may perform the following: respectively carrying out geometric shape detection and geometric relation detection on a plurality of panoramic images of the three-dimensional space by utilizing the geometric prior of the three-dimensional space, and determining a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction; according to the geometric prior of the three-dimensional space, constructing geometric constraints aiming at the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection, and determining the translation vector and the position information of a camera for shooting the panoramic image for three-dimensional reconstruction according to the constructed geometric constraints; and performing three-dimensional reconstruction on the three-dimensional space by using the rotation matrix, the translation vector and the position information.
For example, components such as processor 420 and memory 410 may communicate over a network connection. The network may include a wireless network, a wired network, and/or any combination of wireless and wired networks. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things (Internet of Things) based on the Internet and/or a telecommunications network, and/or any combination thereof, and/or the like. The wired network may communicate by using twisted pair, coaxial cable, or optical fiber transmission, for example, and the wireless network may communicate by using 3G/4G/5G mobile communication network, bluetooth, Zigbee, or WiFi, for example. The present disclosure is not limited herein as to the type and function of the network.
For example, the processor 420 may control other components in the three-dimensional reconstruction device 400 to perform desired functions. The processor 620 may be a device having a data processing capability and/or a program execution capability, such as a Central Processing Unit (CPU), a Tensor Processor (TPU), or a Graphics Processing Unit (GPU). The Central Processing Unit (CPU) may be an X86 or ARM architecture, etc. The GPU may be separately integrated directly onto the motherboard, or built into the north bridge chip of the motherboard. The GPU may also be built into the Central Processing Unit (CPU).
For example, memory 410 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, Random Access Memory (RAM), cache memory (or the like). The non-volatile memory may include, for example, Read Only Memory (ROM), a hard disk, an Erasable Programmable Read Only Memory (EPROM), a portable compact disc read only memory (CD-ROM), USB memory, flash memory, and the like. One or more computer-readable instructions may be stored on a computer-readable storage medium and executed by the processor 420 to implement various functions of the three-dimensional reconstruction apparatus 400. Various application programs and various data and the like can also be stored in the storage medium.
For example, the detailed description of the process of performing the three-dimensional reconstruction method by the three-dimensional reconstruction apparatus 400 may refer to the related description in the embodiment of the three-dimensional reconstruction method applied to the three-dimensional terminal, and repeated parts are not repeated.
Referring now to fig. 5, a schematic diagram of a three-dimensional reconstruction apparatus 500 suitable for implementing embodiments of the present disclosure is shown. The three-dimensional reconstruction apparatus in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a smartphone, a notebook computer, a PAD (tablet computer), and the like, and a stationary terminal such as a desktop computer, and the like. The three-dimensional reconstruction apparatus shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 5, the three-dimensional reconstruction apparatus 500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 510, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)520 or a program loaded from a storage device 580 into a Random Access Memory (RAM) 530. In the RAM 530, various programs and data necessary for the operation of the three-dimensional reconstruction apparatus 500 are also stored. The processing device 510, the ROM 520, and the RAM 530 are connected to each other by a bus 540. An input/output (I/O) interface 550 is also connected to bus 540.
Generally, the following devices may be connected to I/O interface 550: input devices 560 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 570 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, or the like; a storage device 580; and a communication device 590. The storage 580 may include various forms of program storage units as well as data storage units such as magnetic tape, hard disk, Read Only Memory (ROM), Random Access Memory (RAM), which can be used to store various data files used for computer processing and/or communications, as well as possible program instructions executed by the processing device 510. The communication device 590 may allow the three-dimensional reconstruction apparatus 500 to communicate with other devices wirelessly or by wire to exchange data, such as to transmit and receive information and data. While fig. 5 illustrates a three-dimensional reconstruction device 500 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as a computer program. The computer program comprises, for example, program code for performing the method illustrated in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network through the communication device 590, or installed from the storage device 580, or installed from the ROM 520. The computer program, when executed by the processing device 510, performs the above-described functions defined in the remote presentation method of the embodiments of the present disclosure.
It should be noted that the three-dimensional reconstruction device 500 provided in the embodiment of the present disclosure may adopt an Android (Android) system, an IOS system, a Linux system, a Windows system, and the like.
At least one embodiment of the present disclosure further provides a storage medium, and fig. 6 is a schematic diagram of a storage medium provided in an embodiment of the present disclosure.
For example, as shown in FIG. 6, one or more computer readable instructions 601 may be stored non-temporarily on a storage medium 600. For example, the computer readable instructions 601, when executed by a computer, may perform one or more steps in a three-dimensional reconstruction method applied to a three-dimensional space according to the above.
For example, the storage medium 600 may be applied to the three-dimensional reconstruction apparatus 400 described above, and may include the memory 410 in the three-dimensional reconstruction apparatus 400, for example.
For example, the description of the storage medium 600 may refer to the description of the memory in the embodiment of the three-dimensional reconstruction apparatus 400, and repeated descriptions are omitted.
Various changes, substitutions and alterations to the techniques herein may be made without departing from the techniques of the teachings as defined by the appended claims. Moreover, the scope of the claims of the present disclosure is not limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims (22)

1. A method of three-dimensional reconstruction of a three-dimensional space, comprising:
respectively carrying out geometric shape detection and geometric relation detection on a plurality of panoramic images of the three-dimensional space by utilizing the geometric prior of the three-dimensional space, and determining a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction;
according to the geometric prior of the three-dimensional space, constructing geometric constraints for the geometric shapes and the geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection, and determining translation vectors and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints;
performing a three-dimensional reconstruction of the three-dimensional space using the rotation matrix, the translation vector, and the position information.
2. The method of claim 1, wherein the geometric prior of the three-dimensional space comprises at least one of:
two opposite surfaces in the three-dimensional space are parallel to each other, two adjacent surfaces in the three-dimensional space are perpendicular to each other, and three surfaces intersecting at a point in the three-dimensional space are perpendicular to each other pairwise.
3. The method of claim 1, wherein the using a geometric prior of the three-dimensional space to perform geometry detection and geometry relationship detection on the plurality of panoramic images of the three-dimensional space respectively comprises:
correcting the panoramic image into a perspective image;
performing geometry detection on the perspective image, wherein the geometry detection comprises straight line detection;
and determining at least one vanishing point in the three-dimensional space at least according to the result of the straight line detection.
4. The method of claim 3, wherein performing geometry detection and geometry relationship detection on a plurality of panoramic images of the three-dimensional space, respectively, using a geometric prior of the three-dimensional space, and determining a rotation matrix for a camera taking the panoramic images for three-dimensional reconstruction comprises:
determining rotation matrixes of cameras corresponding to at least two perspective images according to the straight line detection results and the determined vanishing points of the at least two perspective images in the plurality of perspective images; or
And correcting the initial rotation matrix obtained by the inertial measurement unit of the camera according to at least the straight line detection result and the determined vanishing point of at least one perspective image in the plurality of perspective images to obtain the rotation matrix of the camera corresponding to the at least one perspective image.
5. The method of claim 3, wherein constructing geometric constraints for the geometry and geometric relationships detected in the geometry detection and the geometric relationship detection, respectively, according to a geometric prior of the three-dimensional space, and determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints comprises:
respectively carrying out feature point detection on a plurality of perspective images, carrying out feature point matching according to the result of the feature point detection, and acquiring at least one straight line on at least two perspective images in the plurality of perspective images;
constructing a co-linear constraint for the at least one straight line, determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction at least from the constructed co-linear constraint.
6. The method of claim 3, wherein the geometry detection and the geometry relationship detection are performed on the plurality of panoramic images of the three-dimensional space, respectively, using a geometric prior of the three-dimensional space, further comprising:
and performing the geometric relationship detection at least according to the result of the straight line detection, wherein the geometric relationship detection comprises at least one of the following steps: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection.
7. The method of claim 6, wherein constructing geometric constraints for the geometry and geometric relationships detected in the geometry detection and the geometric relationship detection, respectively, according to a geometric prior of the three-dimensional space, and determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints comprises:
respectively detecting feature points of a plurality of perspective images, and acquiring at least two straight lines which respectively exist on at least two perspective images of the plurality of perspective images and meet the detected geometric relationship;
and constructing a geometric relation constraint aiming at the at least two straight lines, and determining a translation vector and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction at least according to the constructed geometric relation constraint.
8. The method of any of claims 1-7, further comprising:
and according to the geometric prior of the three-dimensional space, carrying out bundle adjustment on the three-dimensional reconstruction result of the three-dimensional space.
9. The method of claim 8, wherein bundle adjusting results of a three-dimensional reconstruction of the three-dimensional space according to a geometric prior of the three-dimensional space comprises at least one of:
parameterizing at least one straight line obtained in the three-dimensional reconstruction of the three-dimensional space, and performing bundle adjustment by utilizing a collinearity constraint aiming at the at least one straight line;
parameterizing at least two coplanar straight lines obtained in the three-dimensional reconstruction of the three-dimensional space, and performing cluster adjustment by utilizing coplanarity constraint aiming at the at least two coplanar straight lines.
10. The method of any one of claims 1-7, wherein the three-dimensional space is a building interior space comprising a bottom surface and a top surface parallel to each other and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form an at least partially enclosed space.
11. An apparatus for three-dimensional reconstruction of a three-dimensional space, comprising:
a detection unit configured to perform geometry detection and geometry relation detection on a plurality of panoramic images of the three-dimensional space, respectively, using a geometric prior of the three-dimensional space, and determine a rotation matrix of a camera for shooting the panoramic images for three-dimensional reconstruction;
a determination unit configured to construct geometric constraints for the geometric shapes and geometric relationships respectively detected in the geometric shape detection and the geometric relationship detection according to a geometric prior of the three-dimensional space, and determine translation vectors and position information of cameras for shooting the panoramic image for the three-dimensional reconstruction according to the constructed geometric constraints;
a reconstruction unit configured to perform a three-dimensional reconstruction of the three-dimensional space using the rotation matrix, the translation vector, and the position information.
12. The apparatus of claim 11, wherein the geometric prior of the three-dimensional space comprises at least one of:
two opposite surfaces in the three-dimensional space are parallel to each other, two adjacent surfaces in the three-dimensional space are perpendicular to each other, and three surfaces intersecting at a point in the three-dimensional space are perpendicular to each other pairwise.
13. The apparatus of claim 11, wherein,
the detection unit corrects the panoramic image into a perspective image;
performing geometry detection on the perspective image, wherein the geometry detection comprises straight line detection;
and determining at least one vanishing point in the three-dimensional space at least according to the result of the straight line detection.
14. The apparatus of claim 13, wherein,
the detection unit determines rotation matrixes of cameras corresponding to at least two perspective images according to the straight line detection results and the determined vanishing points of the at least two perspective images in the plurality of perspective images; or
And correcting the initial rotation matrix obtained by the inertial measurement unit of the camera according to at least the straight line detection result and the determined vanishing point of at least one perspective image in the plurality of perspective images to obtain the rotation matrix of the camera corresponding to the at least one perspective image.
15. The apparatus of claim 13, wherein,
the determining unit respectively detects the feature points of the plurality of perspective images, matches the feature points according to the detection result of the feature points, and acquires at least one straight line on at least two perspective images in the plurality of perspective images;
constructing a co-linear constraint for the at least one straight line, determining translation vectors and position information of a camera taking the panoramic image for the three-dimensional reconstruction at least from the constructed co-linear constraint.
16. The apparatus of claim 13, wherein,
the detection unit performs the geometric relationship detection at least according to a result of the straight line detection, wherein the geometric relationship detection includes at least one of: coplanar straight line group detection, parallel straight line group detection, collinear straight line group detection and vertical straight line group detection.
17. The apparatus of claim 16, wherein,
the determining unit detects feature points of the plurality of perspective images respectively and acquires at least two straight lines, wherein the at least two straight lines are respectively present on at least two perspective images of the plurality of perspective images and meet the detected geometric relationship;
and constructing a geometric relation constraint aiming at the at least two straight lines, and determining a translation vector and position information of a camera for shooting the panoramic image for the three-dimensional reconstruction at least according to the constructed geometric relation constraint.
18. The apparatus of any of claims 11-17, further comprising:
and the adjusting unit is configured to perform bundle adjustment on the result of the three-dimensional reconstruction of the three-dimensional space according to the geometric prior of the three-dimensional space.
19. The apparatus of claim 18, wherein,
the adjusting unit performs bundle adjustment in at least one of the following ways:
parameterizing at least one straight line obtained in the three-dimensional reconstruction of the three-dimensional space, and performing bundle adjustment by utilizing a collinearity constraint aiming at the at least one straight line;
parameterizing at least two coplanar straight lines obtained in the three-dimensional reconstruction of the three-dimensional space, and performing cluster adjustment by utilizing coplanarity constraint aiming at the at least two coplanar straight lines.
20. The apparatus of any one of claims 11-17, wherein the three-dimensional space is a building interior space comprising a bottom surface and a top surface parallel to each other and a wall surface at least partially between and perpendicular to the bottom surface and the top surface to form an at least partially enclosed space.
21. An apparatus for three-dimensional reconstruction of a three-dimensional space, comprising:
one or more processors; and
one or more memories having computer program instructions stored therein,
wherein the computer program instructions, when executed by the one or more processors, perform the three-dimensional reconstruction method of any one of claims 1-10.
22. A computer-readable storage medium having stored thereon instructions that, when executed by a processor, cause the processor to perform the three-dimensional reconstruction method of any one of claims 1-10.
CN201911312900.9A 2019-12-18 2019-12-18 Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium Active CN111161336B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911312900.9A CN111161336B (en) 2019-12-18 2019-12-18 Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911312900.9A CN111161336B (en) 2019-12-18 2019-12-18 Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium

Publications (2)

Publication Number Publication Date
CN111161336A true CN111161336A (en) 2020-05-15
CN111161336B CN111161336B (en) 2021-01-29

Family

ID=70557315

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911312900.9A Active CN111161336B (en) 2019-12-18 2019-12-18 Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium

Country Status (1)

Country Link
CN (1) CN111161336B (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612842A (en) * 2020-05-29 2020-09-01 贝壳技术有限公司 Method and device for generating pose estimation model
CN112085794A (en) * 2020-09-11 2020-12-15 中德（珠海）人工智能研究院有限公司 Space positioning method and three-dimensional reconstruction method applying same
CN112288853A (en) * 2020-10-29 2021-01-29 字节跳动有限公司 Three-dimensional reconstruction method, three-dimensional reconstruction device, and storage medium
CN113034681A (en) * 2021-04-07 2021-06-25 清华大学 Three-dimensional reconstruction method and device for spatial plane relation constraint
CN113989376A (en) * 2021-12-23 2022-01-28 贝壳技术有限公司 Method and device for acquiring indoor depth information and readable storage medium
CN114926371A (en) * 2022-06-27 2022-08-19 北京五八信息技术有限公司 Vertical correction and vanishing point detection method and device for panorama and storage medium
CN115330942A (en) * 2022-08-11 2022-11-11 北京城市网邻信息技术有限公司 Multilayer space three-dimensional modeling method, device and computer readable storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140333615A1 (en) * 2013-05-11 2014-11-13 Mitsubishi Electric Research Laboratories, Inc. Method For Reconstructing 3D Scenes From 2D Images
CN107256576A (en) * 2017-04-21 2017-10-17 深圳市蜗牛窝科技有限公司 The methods of exhibiting and device of three-dimensional scenic in picture
US20180096527A1 (en) * 2013-10-25 2018-04-05 Appliance Computing III, Inc. Image-based rendering of real spaces
CN109035327A (en) * 2018-06-25 2018-12-18 北京大学 Panorama camera Attitude estimation method based on deep learning
CN110189399A (en) * 2019-04-26 2019-08-30 浙江大学 A kind of method and system that interior three-dimensional layout rebuilds
CN110189402A (en) * 2019-05-22 2019-08-30 武汉尺子科技有限公司 A kind of floor plan three-dimensional display method, medium, equipment and device based on AR

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140333615A1 (en) * 2013-05-11 2014-11-13 Mitsubishi Electric Research Laboratories, Inc. Method For Reconstructing 3D Scenes From 2D Images
US20180096527A1 (en) * 2013-10-25 2018-04-05 Appliance Computing III, Inc. Image-based rendering of real spaces
CN107256576A (en) * 2017-04-21 2017-10-17 深圳市蜗牛窝科技有限公司 The methods of exhibiting and device of three-dimensional scenic in picture
CN109035327A (en) * 2018-06-25 2018-12-18 北京大学 Panorama camera Attitude estimation method based on deep learning
CN110189399A (en) * 2019-04-26 2019-08-30 浙江大学 A kind of method and system that interior three-dimensional layout rebuilds
CN110189402A (en) * 2019-05-22 2019-08-30 武汉尺子科技有限公司 A kind of floor plan three-dimensional display method, medium, equipment and device based on AR

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
YANG HAO，ET AL.: "《Indoor structure understanding from single 360 cylindrical panoramic image》", 《PROCEEDINGS OF THE 2013 INTERNATIONAL CONFERENCE ON COMPUTER-AIDED DESIGN AND COMPUTER GRAPHICS》 *
王海菲 等: "《复杂室内图像的灭点检测与箱体重建方法》", 《计算机科学与探索》 *

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612842A (en) * 2020-05-29 2020-09-01 贝壳技术有限公司 Method and device for generating pose estimation model
CN111612842B (en) * 2020-05-29 2023-08-18 如你所视(北京)科技有限公司 Method and device for generating pose estimation model
CN112085794A (en) * 2020-09-11 2020-12-15 中德（珠海）人工智能研究院有限公司 Space positioning method and three-dimensional reconstruction method applying same
CN112085794B (en) * 2020-09-11 2022-05-17 中德（珠海）人工智能研究院有限公司 Space positioning method and three-dimensional reconstruction method applying same
CN112288853A (en) * 2020-10-29 2021-01-29 字节跳动有限公司 Three-dimensional reconstruction method, three-dimensional reconstruction device, and storage medium
CN112288853B (en) * 2020-10-29 2023-06-20 字节跳动有限公司 Three-dimensional reconstruction method, three-dimensional reconstruction device, and storage medium
CN113034681A (en) * 2021-04-07 2021-06-25 清华大学 Three-dimensional reconstruction method and device for spatial plane relation constraint
CN113034681B (en) * 2021-04-07 2022-05-03 清华大学 Three-dimensional reconstruction method and device for spatial plane relation constraint
CN113989376A (en) * 2021-12-23 2022-01-28 贝壳技术有限公司 Method and device for acquiring indoor depth information and readable storage medium
CN114926371A (en) * 2022-06-27 2022-08-19 北京五八信息技术有限公司 Vertical correction and vanishing point detection method and device for panorama and storage medium
CN115330942A (en) * 2022-08-11 2022-11-11 北京城市网邻信息技术有限公司 Multilayer space three-dimensional modeling method, device and computer readable storage medium
CN115330942B (en) * 2022-08-11 2023-03-28 北京城市网邻信息技术有限公司 Multilayer space three-dimensional modeling method, device and computer readable storage medium

Also Published As

Publication number Publication date
CN111161336B (en) 2021-01-29

Similar Documents

Publication Publication Date Title
CN111161336B (en) Three-dimensional reconstruction method, three-dimensional reconstruction apparatus, and computer-readable storage medium
CN111127655B (en) House layout drawing construction method and device, and storage medium
US11200734B2 (en) Method for reconstructing three-dimensional space scene based on photographing
US11270460B2 (en) Method and apparatus for determining pose of image capturing device, and storage medium
US20210233275A1 (en) Monocular vision tracking method, apparatus and non-transitory computer-readable storage medium
CN110383343B (en) Inconsistency detection system, mixed reality system, program, and inconsistency detection method
US10026218B1 (en) Modeling indoor scenes based on digital images
US11557083B2 (en) Photography-based 3D modeling system and method, and automatic 3D modeling apparatus and method
US8787700B1 (en) Automatic pose estimation from uncalibrated unordered spherical panoramas
CN108335353A (en) Three-dimensional rebuilding method, device and system, server, the medium of dynamic scene
US10290049B1 (en) System and method for multi-user augmented reality shopping
CN111145294B (en) Two-dimensional house type graph construction method and device and storage medium
US20180182163A1 (en) 3d model generating system, 3d model generating method, and program
US20210201524A1 (en) Method for generating roof outlines from lateral images
US10600202B2 (en) Information processing device and method, and program
JP2023516656A (en) Efficient localization based on multiple feature types
US8509522B2 (en) Camera translation using rotation from device
WO2019155903A1 (en) Information processing device and method
JP2016114445A (en) Three-dimensional position calculation device, program for the same, and cg composition apparatus
JP6152888B2 (en) Information processing apparatus, control method and program thereof, and information processing system, control method and program thereof
CN114452646A (en) Virtual object perspective processing method and device and computer equipment
US20230334810A1 (en) Methods, storage media, and systems for generating a three-dimensional coordinate system
CA3102860C (en) Photography-based 3d modeling system and method, and automatic 3d modeling apparatus and method
CN117788710A (en) Three-dimensional space fusion method and device based on fish eye pattern, electronic equipment and medium
CN115018877A (en) Special effect display method, device and equipment for ground area and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant