CN113920270B - Layout reconstruction method and system based on multi-view panorama - Google Patents
Layout reconstruction method and system based on multi-view panorama Download PDFInfo
- Publication number
- CN113920270B CN113920270B CN202111527178.8A CN202111527178A CN113920270B CN 113920270 B CN113920270 B CN 113920270B CN 202111527178 A CN202111527178 A CN 202111527178A CN 113920270 B CN113920270 B CN 113920270B
- Authority
- CN
- China
- Prior art keywords
- information
- layout
- panoramic image
- indoor scene
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000013528 artificial neural network Methods 0.000 claims abstract description 17
- 238000012937 correction Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 230000003287 optical effect Effects 0.000 claims description 5
- 230000004927 fusion Effects 0.000 claims description 4
- 230000006872 improvement Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 3
- 238000005034 decoration Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 241000764238 Isis Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 101150050759 outI gene Proteins 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G06T3/08—
Abstract
The invention relates to the technical field of indoor scene layout, in particular to a layout reconstruction method and a system thereof based on multi-view panorama; the method comprises the steps of firstly projecting a panoramic image onto a cube to obtain perspective images of six surfaces, and extracting image characteristics and matching the characteristics from the perspective images of the six surfaces so as to obtain position information and posture information of each panoramic image; then predicting three-dimensional layout information of an indoor scene for each panoramic image by using a pre-trained deep neural network; finally, fusing the position information and the posture information of the panoramic image with the three-dimensional layout information of the indoor scene, thereby completing the reconstruction of the indoor scene; the invention can complete the layout reconstruction of the whole indoor scene only by one panoramic sensor without manual intervention, and can efficiently reconstruct the indoor scene.
Description
Technical Field
The invention relates to the technical field of indoor scene layout, in particular to a layout reconstruction method and a system based on multi-view panorama.
Background
Compared with the reconstruction of an indoor three-dimensional model, the indoor scene layout reconstruction result is simpler and more compact, and the method has a wider application range in the VR/AR field.
The patent of the invention discloses a Chinese patent with the name of an indoor three-dimensional layout reconstruction method (patent number ZL 201910343315), which discloses acquiring image sequence data and inertia measurement data of an indoor scene, and simultaneously performing real-time three-dimensional reconstruction on the indoor scene to obtain a key frame image sequence and a real-time reconstructed position and posture of the key frame image sequence; then, performing off-line three-dimensional reconstruction by using the key frame image sequence and the corresponding real-time reconstructed position and posture to obtain dense three-dimensional point cloud of the indoor scene; then extracting a plane structure from the dense three-dimensional point cloud of the indoor scene, screening and classifying the plane structure to obtain a roof plane, a ground plane and a candidate wall plane, and simultaneously obtaining the floor height; then, constructing a three-dimensional layout of the indoor scene according to a roof plane, a ground plane, a candidate wall plane and a floor height; finally, outputting a three-dimensional layout reconstruction result; the method mainly comprises the steps of extracting a plane structure from dense three-dimensional point cloud of an indoor scene after completing indoor scene reconstruction by utilizing RGB color images, and screening and classifying the plane structure so as to obtain layout information of the indoor scene.
The Chinese patent application with the patent name of an indoor reconstruction method, a device, equipment and a medium (with the patent number of CN 201711163966) discloses acquiring a panoramic image, depth of field data and an acquisition position of indoor decoration; according to the panoramic image, the field depth data and the acquisition position, three-dimensional reconstruction is carried out on the three-dimensional space of the house where the indoor decoration is located, and a three-dimensional house model is generated; the method mainly utilizes a single RGB image to estimate indoor three-dimensional layout information based on a deep neural network, and only can perform layout reconstruction on a small area each time.
Disclosure of Invention
The invention mainly solves the technical problem of providing a layout reconstruction method based on multi-view panorama, which can complete the layout reconstruction of the whole indoor scene only by one panorama sensor without manual intervention and can efficiently reconstruct the indoor scene; a layout reconstruction system based on multi-view panorama is also provided.
In order to solve the technical problems, the invention adopts a technical scheme that: a layout reconstruction method based on multi-view panorama is provided, wherein the method comprises the following steps:
step S1, projecting the panoramic image onto a cube to obtain perspective images of six faces, and extracting image features and matching the features from the perspective images of the six faces to obtain position information and posture information of each panoramic image;
step S2, training each panoramic image by the pre-trained deep neural network, and predicting three-dimensional layout information of an indoor scene;
and step S3, fusing the position information and the posture information of the panoramic image with the three-dimensional layout information of the indoor scene, thereby completing the reconstruction of the indoor scene.
As a modification of the present invention, in step S1, the image feature is extracted using a region with a larger change in gradation as a feature point.
As a further improvement of the present invention, in step S1, feature matching of feature points is performed by optical flow matching.
As a further improvement of the present invention, in step S1, the position information and the orientation information of each panoramic image are calculated based on the extracted image features and feature matching points.
As a further improvement of the present invention, in step S2, when the pre-trained deep neural network is trained for each panoramic image, the panoramic image is input, and the house corner points and the house wall map are output.
As a further improvement of the present invention, in step S2, house corner points, house corner point depths, and two-dimensional house wall lines are extracted from the panoramic image, and then back-projected into a three-dimensional space, so as to predict three-dimensional layout information of an indoor scene.
As a further improvement of the present invention, in step S3, scale information of sparse points is acquired based on the pose information of the panoramic image, and thus the scale information of the three-dimensional layout information of the indoor scene in step S2 is corrected.
As a further improvement of the present invention, in step S3, the scale information of the sparse points is compared with the scale information of the three-dimensional layout information of the indoor scene to obtain a scale correction factor, so that the three-dimensional layout information of the indoor scene is scaled according to the scale correction factor.
As a further improvement of the present invention, in step S3, the three-dimensional layouts of the corrected indoor scenes generated from all the panoramic images are merged to obtain the layout of the indoor scene.
A multi-view panorama based layout reconstruction system, comprising:
the attitude acquisition module is used for projecting the panoramic image onto a cube, acquiring perspective images of six surfaces, and extracting image features and matching the features from the perspective images of the six surfaces so as to obtain position information and attitude information of each panoramic image;
the training layout module is used for training each panoramic image by using a pre-trained deep neural network and predicting three-dimensional layout information of an indoor scene;
and the fusion reconstruction module fuses the position information and the posture information of the panoramic image and the three-dimensional layout information of the indoor scene so as to complete the reconstruction of the indoor scene.
The beneficial effects of the invention are: the method comprises the steps of firstly projecting a panoramic image onto a cube to obtain perspective images of six surfaces, and extracting image characteristics and matching the characteristics from the perspective images of the six surfaces so as to obtain position information and posture information of each panoramic image; then, training each panoramic image by using a pre-trained deep neural network, and predicting three-dimensional layout information of an indoor scene; finally, fusing the position information and the attitude information of the panoramic image with the three-dimensional layout information of the indoor scene, thereby completing the reconstruction of the indoor scene; the invention can complete the layout reconstruction of the whole indoor scene only by one panoramic sensor without manual intervention, and can efficiently reconstruct the indoor scene.
Drawings
FIG. 1 is a block diagram of the steps of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
Referring to fig. 1, a layout reconstruction method based on multi-view panorama according to the present invention includes the following steps:
step S1, projecting the panoramic image onto a cube to obtain perspective images of six surfaces, and extracting image characteristics and matching the characteristics from the perspective images of the six surfaces to obtain position information and posture information of each panoramic image;
step S2, training each panoramic image by the pre-trained deep neural network, and predicting three-dimensional layout information of an indoor scene;
and step S3, fusing the position information and the posture information of the panoramic image with the three-dimensional layout information of the indoor scene, thereby completing the reconstruction of the indoor scene.
The invention can complete the layout reconstruction of the whole indoor scene only by one panoramic sensor without manual intervention, and can efficiently reconstruct the indoor scene.
In step S1, extracting image features using an area with a larger change in gradation as a feature point, and performing feature matching of the feature point by optical flow matching; and calculating the position information and the attitude information of each panoramic image according to the extracted image features and the feature matching points.
Specifically, compared with a perspective image of a scene, the panoramic image has a 360-degree FOV, so that the panoramic image is less susceptible to interference of occlusion and weak/repeated textures when pose estimation is performed in an indoor scene; the real-time pose estimation process of the panoramic image comprises the steps of firstly projecting the panoramic image onto a cube to obtain 6 perspective images, then carrying out image feature extraction and feature matching on the 6 perspective images, and further calculating the position and posture information of each panoramic image, namely, when the panoramic camera carries out feature extraction and feature matching, in order to ensure the calculation efficiency, using an efficient area with large gray scale change as a feature point, and carrying out feature point matching through optical flow, therefore, the selection basis of the feature point is as follows, and the feature point is recorded on the panoramic image
The luminance at is greater than the threshold value than the difference in pixel luminance within a window size of 2m +1,
while performing optical flow matching, assuming that the image is
Is a function associated with time
And further converting the matching problem into an optimization problem to complete feature matching.
There is a need to optimize the formula, wherein
For the unknown to be solved for,
finally, based on the feature matching points and the multi-view solid geometry, the position posture of the panoramic image is adjusted
And the three-dimensional position of the feature point
As an unknown quantity, the solution is made by the following equation, wherein
Representing characteristic points
In the image
The coordinates of the pixel of (a) above,
in step S2, when training each panoramic image with the pre-trained deep neural network, inputting the panoramic image and outputting the panoramic image as house corner points and house wall line map; during operation, house corner points, house corner point depth and two-dimensional house wall lines are extracted from the panoramic image and then are back projected into a three-dimensional space, so that three-dimensional layout information of indoor scenes is predicted.
Specifically, three-dimensional layout information of an indoor scene is directly predicted on each panoramic image by using a deep neural network, and the layout information of the indoor scene corresponding to the current panoramic image can be acquired; the core element of panoramic layout generation is to extract the corner points of the house from the panoramic image
And depth and two-dimensional house wall line thereof
And then back projected into three-dimensional space.
Therefore, a deep neural network needs to be constructed, wherein the input of the deep neural network is a panoramic image, the output of the deep neural network is a house corner point and a house wall line graph, and the layout acquired correspondingly is called as the layout
Wherein
The expression is a neural network, which is,
and
the prediction result of the network is as follows:
in step S3, obtaining scale information of sparse points according to the pose information of the panoramic image, thereby correcting the scale information of the three-dimensional layout information of the indoor scene in step S2; that is, comparing the scale information of the sparse points with the scale information of the three-dimensional layout information of the indoor scene to obtain a scale correction factor, and scaling the three-dimensional layout information of the indoor scene according to the scale correction factor; and performing topological fusion on the three-dimensional layouts of the corrected indoor scenes generated by all the panoramic images to obtain the layouts of the indoor scenes.
Specifically, a single panoramic image and layout information thereof can only complete reconstruction of a part of indoor scenes, but cannot complete three-dimensional reconstruction of the whole indoor scene, and aiming at the problem, multi-view layout reconstruction can be performed by two steps:
firstly, correcting the scale information of indoor layout information by using the scale information of sparse points acquired during real-time pose estimation, and ensuring that the layout generated by a multi-view panoramic image has the same scale; that is, directly comparing the sparse point scale and the corresponding layout scale, and calculating the scale correction factor by the optimization algorithm, for the panoramic image
And corresponding thereto
First, can acquire
Projected coordinates on a panorama
And depth
(ii) a At the same time, the panorama can be laid out
Is projected to
Obtaining
Depth of the site
To eliminate
The following equation can be constructed to solve the scale correction factor
：
And secondly, after the scale correction is finished, fusing layout information generated by a plurality of images.
Because the indoor layout can be simplified into a two-dimensional plane except for the height, a two-dimensional voxel with the resolution ratio of r is constructed firstly, then the layout of a plurality of frames of panoramic images is projected into the voxel, all layout information is further fused according to the voxel, and finally the voxel information is converted into topology information to be finally output, so that the layout reconstruction result of the whole indoor scene is obtained.
The invention also provides a layout reconstruction system based on multi-view panorama, comprising:
the attitude acquisition module is used for projecting the panoramic image onto a cube, acquiring perspective images of six surfaces, and extracting image characteristics and matching the characteristics from the perspective images of the six surfaces so as to obtain position information and attitude information of each panoramic image;
the training layout module is used for training each panoramic image by using a pre-trained deep neural network and predicting three-dimensional layout information of an indoor scene;
and the fusion reconstruction module fuses the position information and the posture information of the panoramic image and the three-dimensional layout information of the indoor scene so as to complete the reconstruction of the indoor scene.
Compared with other indoor layout reconstruction schemes, the indoor layout reconstruction method can complete the layout reconstruction of the whole indoor scene only by one panoramic sensor without manual intervention, and can efficiently reconstruct the indoor scene.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (7)
1. A layout reconstruction method based on multi-view panorama is provided, which is characterized by further comprising the following steps:
step S1, projecting the panoramic image onto a cube to obtain perspective images of six faces, and extracting image features and matching the features from the perspective images of the six faces to obtain position information and posture information of each panoramic image;
step S2, predicting each panoramic image by using a pre-trained deep neural network to obtain three-dimensional layout information of an indoor scene;
step S3, fusing the position information and the posture information of the panoramic image with the three-dimensional layout information of the indoor scene, thereby completing the reconstruction of the indoor scene;
in step S1, image features are extracted using the regions with changed gray levels as feature points;
in step S1, feature matching of the feature points is performed by optical flow matching;
in step S1, calculating position information and orientation information of each panoramic image based on the extracted image features and feature matching points;
the region selection basis of the feature points is positioned in the panoramic imageIAbove (A)x,y) The brightness of the spot is larger than the window size by 2mThe difference in pixel brightness within +1 is greater than a thresholdtTo select, the expression is:
where m is the projection coordinate.
2. The layout reconstruction method based on multi-view panorama of claim 1, wherein in step S2, when a pre-trained deep neural network is used to predict each panoramic image, the input is panoramic image, and the output is house corner points and house wall line map.
3. The layout reconstruction method based on the multi-view panorama of claim 2, wherein in step S2, house corner points, house corner point depths, and two-dimensional house wall lines are extracted from the panoramic image and then back projected into a three-dimensional space, thereby predicting three-dimensional layout information of an indoor scene.
4. The method of claim 3, wherein in step S3, the scale information of sparse points is obtained according to the pose information of the panoramic image, so as to correct the scale information of the three-dimensional layout information of the indoor scene in step S2.
5. The method for reconstructing a layout based on multi-view panorama of claim 4, wherein in step S3, the scale information of the sparse points is compared with the scale information of the three-dimensional layout information of the indoor scene to obtain the scale correction factor, so as to scale the three-dimensional layout information of the indoor scene according to the scale correction factor.
6. The method for reconstructing a layout based on multi-view panorama of claim 5, wherein in step S3, the three-dimensional layout of the corrected indoor scene generated by all the panorama images is merged to obtain the layout of the indoor scene.
7. A multi-view panorama based layout reconstruction system employing the layout reconstruction method according to any one of claims 1 through 6, comprising:
the attitude acquisition module is used for projecting the panoramic image onto a cube, acquiring perspective images of six surfaces, and extracting image features and matching the features from the perspective images of the six surfaces so as to obtain position information and attitude information of each panoramic image;
the training layout module is used for predicting each panoramic image by using a pre-trained deep neural network to acquire three-dimensional layout information of an indoor scene;
and the fusion reconstruction module fuses the position information and the posture information of the panoramic image and the three-dimensional layout information of the indoor scene so as to complete the reconstruction of the indoor scene.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111527178.8A CN113920270B (en) | 2021-12-15 | 2021-12-15 | Layout reconstruction method and system based on multi-view panorama |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111527178.8A CN113920270B (en) | 2021-12-15 | 2021-12-15 | Layout reconstruction method and system based on multi-view panorama |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113920270A CN113920270A (en) | 2022-01-11 |
| CN113920270B true CN113920270B (en) | 2022-08-19 |
Family
ID=79249029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111527178.8A Active CN113920270B (en) | 2021-12-15 | 2021-12-15 | Layout reconstruction method and system based on multi-view panorama |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113920270B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114449247B (en) * | 2022-04-11 | 2022-07-22 | 深圳市其域创新科技有限公司 | Multi-channel video 3D (three-dimensional) superposition method and system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11074701B2 (en) * | 2019-12-13 | 2021-07-27 | Reconstruct Inc. | Interior photographic documentation of architectural and industrial environments using 360 panoramic videos |
| CN111275750B (en) * | 2020-01-19 | 2022-05-13 | 武汉大学 | Indoor space panoramic image generation method based on multi-sensor fusion |
| CN111462311B (en) * | 2020-03-31 | 2023-10-24 | 北京小米松果电子有限公司 | Panorama generation method and device and storage medium |
| CN111797748A (en) * | 2020-06-28 | 2020-10-20 | 北京比邻弘科科技有限公司 | Image recognition method based on deep learning |
| CN113298928A (en) * | 2021-03-26 | 2021-08-24 | 阿里巴巴新加坡控股有限公司 | House three-dimensional reconstruction method, device, equipment and storage medium |
| CN113409442A (en) * | 2021-05-13 | 2021-09-17 | 杭州电子科技大学 | Method for fusing multi-panorama and reconstructing three-dimensional image |
| CN113487723B (en) * | 2021-06-23 | 2023-04-18 | 武汉微景易绘科技有限公司 | House online display method and system based on measurable panoramic three-dimensional model |
-
2021
- 2021-12-15 CN CN202111527178.8A patent/CN113920270B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113920270A (en) | 2022-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111325794B (en) | Visual simultaneous localization and map construction method based on depth convolution self-encoder | |
| CN107578436B (en) | Monocular image depth estimation method based on full convolution neural network FCN | |
| CN108876814B (en) | Method for generating attitude flow image | |
| CN110223370B (en) | Method for generating complete human texture map from single-view picture | |
| WO2015188684A1 (en) | Three-dimensional model reconstruction method and system | |
| US20170213320A1 (en) | Reconstruction of articulated objects from a moving camera | |
| KR102152432B1 (en) | A real contents producing system using the dynamic 3D model and the method thereof | |
| CN113077505B (en) | Monocular depth estimation network optimization method based on contrast learning | |
| Zhang et al. | Critical regularizations for neural surface reconstruction in the wild | |
| CN112270736A (en) | Augmented reality processing method and device, storage medium and electronic equipment | |
| CN107610219A (en) | The thick densification method of Pixel-level point cloud that geometry clue perceives in a kind of three-dimensional scenic reconstruct | |
| CN115298708A (en) | Multi-view neural human body rendering | |
| CN115239870A (en) | Multi-view stereo network three-dimensional reconstruction method based on attention cost body pyramid | |
| CN114996814A (en) | Furniture design system based on deep learning and three-dimensional reconstruction | |
| CN114494589A (en) | Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and computer-readable storage medium | |
| CN116468769A (en) | Depth information estimation method based on image | |
| CN113920270B (en) | Layout reconstruction method and system based on multi-view panorama | |
| CN116051747A (en) | House three-dimensional model reconstruction method, device and medium based on missing point cloud data | |
| CN111640172A (en) | Attitude migration method based on generation of countermeasure network | |
| Jiang et al. | H $ _ {2} $-Mapping: Real-time Dense Mapping Using Hierarchical Hybrid Representation | |
| CN112927348B (en) | High-resolution human body three-dimensional reconstruction method based on multi-viewpoint RGBD camera | |
| CN112002007B (en) | Model acquisition method and device based on air-ground image, equipment and storage medium | |
| Reid et al. | Multiview segmentation and tracking of dynamic occluding layers | |
| CN117274514A (en) | Remote sensing image generation method and device based on ground-air visual angle geometric transformation | |
| CN114935316B (en) | Standard depth image generation method based on optical tracking and monocular vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information |
Inventor after: Yan Qingsong Inventor after: Zhao Kaiyong Inventor before: Yan Qingsong |
|
| CB03 | Change of inventor or designer information |