WO2022011898A1 - 一种三维重建方法、装置、设备及计算机可读存储介质 - Google Patents

一种三维重建方法、装置、设备及计算机可读存储介质 Download PDF

Info

Publication number
WO2022011898A1
WO2022011898A1 PCT/CN2020/127362 CN2020127362W WO2022011898A1 WO 2022011898 A1 WO2022011898 A1 WO 2022011898A1 CN 2020127362 W CN2020127362 W CN 2020127362W WO 2022011898 A1 WO2022011898 A1 WO 2022011898A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature
interest
target
current
circular
Prior art date
Application number
PCT/CN2020/127362
Other languages
English (en)
French (fr)
Inventor
尚家乐
姜滨
迟小羽
Original Assignee
歌尔股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 歌尔股份有限公司 filed Critical 歌尔股份有限公司
Priority to US18/004,847 priority Critical patent/US20230306619A1/en
Publication of WO2022011898A1 publication Critical patent/WO2022011898A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20072—Graph-based image processing

Definitions

  • the present application relates to the field of head-mounted display devices, and in particular, to a three-dimensional reconstruction method, apparatus, device, and computer-readable storage medium.
  • VR Virtual Reality, virtual reality
  • AR Augmented Reality, augmented reality
  • MR Mediated Reality, mixed reality
  • SLAM Simultaneous Reality
  • the head-mounted display device In the prior art, in most application scenarios, the head-mounted display device often only needs to reconstruct the simple geometric structure of the complex scene to meet the user's positioning and virtual-real interaction requirements.
  • the 3D reconstruction process Due to the large amount of feature information in the real scene, Therefore, the 3D reconstruction process has the disadvantages of time-consuming reconstruction and poor reconstruction effect due to excessive redundant feature information and poor feature quality, which seriously restricts the application of head-mounted display devices.
  • the purpose of the present application is to provide a three-dimensional reconstruction method, apparatus, device and computer-readable storage medium, so as to improve the efficiency and accuracy of three-dimensional reconstruction of regular objects by a head-mounted display device, and improve user experience.
  • the present application provides a three-dimensional reconstruction method, including:
  • the respective corresponding features in each of the to-be-processed windows include point features and/or line feature constraint points.
  • the target to-be-processed window is a to-be-processed window with corresponding features, the number of circular regions of interest corresponding to each of the target to-be-processed windows is greater than a threshold, and each of the target to-be-processed windows corresponds to
  • the circular region of interest includes a first circular region of interest or the first circular region of interest and a second circular region of interest, and the first circular region of interest is a circle in which only one of the features exists
  • the second circular region of interest is a circular region of interest in which the line feature constraint points exist.
  • performing feature screening on the features in the target circular region of interest to obtain target features corresponding to each of the target circular regions of interest including:
  • the target feature corresponding to each target circular region of interest is determined; wherein, each target The respective target features corresponding to the circular regions of interest include the target point feature, or the target point feature and the line feature constraint point.
  • performing point feature extraction and line feature extraction on the to-be-processed window to determine the respective corresponding features in each of the to-be-processed windows including:
  • the circular quadtree iterative splitting is performed on each target to-be-processed window to obtain a circular region of interest corresponding to each of the target to-be-processed windows, including:
  • the current circular region of interest is used as the circular region of interest corresponding to the current target pending window
  • the current circular region of interest is used as the circular region of interest corresponding to the current target to-be-processed window;
  • the current circular region of interest is not the first circular region of interest, then determine whether the number of circular regions of interest in the current target to-be-processed window is greater than the threshold;
  • the current circular region of interest is used as the circular region of interest corresponding to the current target to-be-processed window
  • the feature screening is performed on the point features in the target circular region of interest to obtain a preset number of target point features corresponding to each of the target circular regions of interest.
  • one feature point is selected from all feature points in the current target circular region of interest as the target point feature corresponding to the current target circular region of interest.
  • the grayscale gradient of each feature point in the current target circular region of interest is compared with the grayscale gradient of the sampling point to obtain the current target circle.
  • the binary feature quality value of each feature point in the region of interest including:
  • using the target feature to perform three-dimensional reconstruction corresponding to the current image frame including:
  • the three-dimensional reconstruction corresponding to the current image frame is improved; wherein, the non-target circular region of interest is each of the A circular area of interest other than the target circular area of interest in the circular area of interest corresponding to the target to-be-processed window.
  • using the target feature to perform three-dimensional reconstruction corresponding to the current image frame including:
  • feature matching is performed between each feature to be matched and the previous image frame, and a feature matching result of each feature to be matched is obtained; wherein, the feature to be matched includes the target feature;
  • the to-be-matched features in the three-dimensional reconstruction feature set are divided into three-dimensional edge and corner reference features and three-dimensional surface reference features, and the three-dimensional edge and corner reference features and the three-dimensional surface reference features are used to perform three-dimensional reconstruction corresponding to the current image frame; wherein , the three-dimensional edge and corner reference feature is the feature to be matched that satisfies the linear constraint relationship of the corresponding line feature.
  • the binary feature quality value of each feature to be matched is used to perform feature matching on each feature to be matched with the previous image frame, and each feature to be matched is obtained.
  • Describe the feature matching results of the features to be matched including:
  • the target descriptor is the sum of the first descriptor and the second descriptor, and the first descriptor
  • the sub is a binary descriptor calculated by the LBD algorithm for the line feature corresponding to the current feature to be matched, and the second descriptor is the binary feature quality value of the current feature to be matched.
  • the application also provides a three-dimensional reconstruction device, including:
  • the sliding calculation module is used to perform sliding calculation on the acquired current image frame according to the preset window size to obtain the to-be-processed window;
  • a feature extraction module configured to perform point feature extraction and line feature extraction on the to-be-processed window, and determine the respective corresponding features in each of the to-be-processed windows;
  • the iterative splitting module is used for performing circular quadtree iterative splitting on each target to-be-processed window to obtain a circular region of interest corresponding to each of the target to-be-processed windows;
  • a feature screening module configured to perform feature screening on the features in the target circular region of interest to obtain target features corresponding to each of the target circular regions of interest
  • the 3D reconstruction module is used for performing 3D reconstruction corresponding to the current image frame by using the target feature.
  • the application also provides a three-dimensional reconstruction device, including:
  • the processor is configured to implement the steps of the above-mentioned three-dimensional reconstruction method when executing the computer program.
  • the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method described above are implemented.
  • a three-dimensional reconstruction method provided by the present application includes: according to a preset window size, performing sliding calculation on the acquired current image frame to obtain a to-be-processed window; performing point feature extraction and line feature extraction on the to-be-processed window, and determining The respective corresponding features in each of the to-be-processed windows; iteratively split a circular quadtree for each target to-be-processed window to obtain a circular region of interest corresponding to each of the target to-be-processed windows; Performing feature screening on the features in the region of interest to obtain target features corresponding to each of the target circular regions of interest; using the target features to perform three-dimensional reconstruction corresponding to the current image frame;
  • the present application introduces line features in the feature extraction stage of 3D reconstruction, so that the line segment features in the 3D reconstruction scene can be better perceived and the reconstruction efficiency of simple and regular 3D models is accelerated;
  • the area of interest is extracted, and the feature extraction is performed by window sliding, and the features in the 3D reconstruction are homogenized and de-redundant in the form of a quadtree, thereby improving the efficiency and accuracy of the 3D reconstruction and improving the user experience.
  • the present application also provides a three-dimensional reconstruction device, device and computer-readable storage medium, which also have the above beneficial effects.
  • FIG. 1 is a flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application
  • FIG. 2 is a flowchart of another three-dimensional reconstruction method provided by an embodiment of the present application.
  • FIG. 3 is a schematic flowchart of another three-dimensional reconstruction method provided by an embodiment of the present application.
  • FIG. 4 is a display diagram of a line feature constraint point of a line feature provided by an embodiment of the present application.
  • FIG. 5 is a display diagram of a circular quadtree iterative splitting provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a circular quadtree iterative splitting provided by an embodiment of the present application.
  • FIG. 7 is a flowchart of iterative splitting of a circular quadtree provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of sampling of a circular region of interest provided by an embodiment of the present application.
  • FIG. 9 is a structural block diagram of a three-dimensional reconstruction apparatus provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a three-dimensional reconstruction device according to an embodiment of the present application.
  • FIG. 1 is a flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application.
  • the method can include:
  • Step 101 According to the preset window size, perform sliding calculation on the acquired current image frame to obtain the to-be-processed window.
  • this step may be for the processor in the 3D reconstruction device (such as VR device, AR device, MR device, etc.) to extract a window with a preset window size (that is, a to-be-processed window) on the current image frame.
  • a preset window size that is, a to-be-processed window
  • the number of windows of the window to be processed obtained in this step may be determined by the size of the current image frame and the preset window size.
  • the preset window size is a preset window length
  • the specific relationship may be as follows:
  • N can be the number of windows to be processed;
  • L can be the length of the current image frame;
  • l can be the length of the cropping area of the current image frame, and the setting of l can avoid edge distortion in the feature extraction process.
  • Feature extraction error; l' can be a preset window length.
  • Step 102 Perform point feature extraction and line feature extraction on the window to be processed, and determine the corresponding feature in each window to be processed.
  • the processor may perform point feature extraction and line feature extraction on the to-be-processed window after extracting the to-be-processed window from the current image frame, and extract the point features and line features in the to-be-processed window.
  • Data corresponding to the feature (ie, feature); that is, each window to be processed determined in this step may include multiple features.
  • the specific content of the corresponding features in each window to be processed extracted in this step can be set by the designer according to the usage scenario and user requirements, such as point feature extraction and line feature extraction only for the window to be processed.
  • the respective corresponding features in each window to be processed may include point features and/or line features, that is, each feature may be a point feature or a line feature.
  • the line feature constraint point corresponding to the obtained line feature can also be extracted, and the obtained line feature constraint point and point feature are used as the feature corresponding to the to-be-processed window, that is,
  • the corresponding features in each window to be processed may include point features and/or line feature constraint points corresponding to line features, and each feature may be a point feature or a line feature constraint point, so that the line feature constraint point corresponding to the line feature is extracted by extracting the line feature constraint point.
  • This embodiment does not impose any limitation on this.
  • this embodiment does not limit the specific way of performing step 101 and this step.
  • the processor can extract a current window to be processed of the current image frame by means of window sliding, and then perform point feature extraction on the current window to be processed. and line feature extraction, determine the corresponding features in the current to-be-processed window, and then extract the next to-be-processed window; the processor may also extract all the to-be-processed windows of the current image frame, and then use this step to extract each to-be-processed window. Perform point feature extraction and line feature extraction to determine the corresponding features in each window to be processed.
  • Step 103 Perform iterative splitting of a circular quadtree on each target to-be-processed window to obtain a circular region of interest corresponding to each target to-be-processed window.
  • the target to-be-processed window in this step may be a to-be-processed window that needs to be iteratively split by the circular quadtree.
  • the target to-be-processed window can be all the to-be-processed windows, that is, this step can be performed on each to-be-processed window corresponding to the current image frame.
  • the circular quadtree is iteratively split;
  • the target to-be-processed window may also be a to-be-processed window with corresponding features, that is, in this step, a circular quad-tree iterative split can be performed on the to-be-processed window with the corresponding feature extracted in step 102;
  • the to-be-processed window may also be a to-be-processed window in which the number of features is greater than the feature threshold and/or the required features (eg, line feature constraint points) exist. This embodiment does not impose any limitation on this.
  • this step may be to iteratively split the circular quadtree shown in FIG. 5 for each target pending window by the processor, and use the quadtree structure shown in Dimension image frame area is iteratively divided, and at the same time, circular area (ie circular area of interest) is used to divide features, which can more accurately perceive the gray gradient information of features in the divided quadtree space.
  • the specific content of the circular region of interest corresponding to each target to-be-processed window obtained in this step can be determined by the designer Set according to practical scenarios and user needs.
  • the number of circular regions of interest corresponding to each target window to be processed can be greater than the threshold, and each circular region of interest corresponding to the target window to be processed can contain features, and each The circular region of interest corresponding to the target to-be-processed window may include a first circular region of interest or a first circular region of interest and a second circular region of interest; wherein the first circular region of interest may be a feature
  • the number of circular ROIs is less than or equal to the preset number.
  • the preset number is 1
  • the first circular ROI can be a circular ROI with only one feature
  • the second circular ROI can be It is a circular region of interest where there are features corresponding to line features (such as line feature constraint points). This embodiment does not impose any limitation on this.
  • Step 104 Perform feature screening on the features in the target circular region of interest to obtain target features corresponding to each target circular region of interest.
  • the target circular region of interest in this step may be a circular region of interest that needs to be screened for features.
  • the target circular area of interest can be a circular area of interest with more than a preset number of point features; that is, That is to say, the feature screening in this step can only perform feature screening on the point features in the target circular region of interest, so as to obtain a preset number of selected point features (ie, target point features) in each target circular region of interest.
  • this step may include feature screening on the point features in the target circular region of interest to obtain a preset number of target point features in each target circular region of interest;
  • the target point feature and the line feature constraint point determine the target feature corresponding to each target circular region of interest; wherein, the target feature corresponding to each target circular region of interest includes the target point feature, or the target point feature and the line Feature constraint points.
  • the target circular ROI can also be a circular ROI containing line feature constraint points and point features, and a circular ROI containing more than a preset number of point features and not containing line feature constraint points; that is,
  • the feature screening in this step can filter out the point features in the target circular region of interest containing the line feature constraint points, and perform feature screening on the point features in the target circular region of interest containing more than a preset number of point features, A pre-screened preset number of point features in the target circular region of interest is obtained, so that the target feature corresponding to a target circular region of interest may be an existing line feature or a preset number of point features.
  • This embodiment does not impose any limitation on this.
  • Step 105 Using the target feature, perform three-dimensional reconstruction corresponding to the current image frame.
  • this step may be to use the obtained target features corresponding to each target circular region of interest to perform three-dimensional reconstruction corresponding to the current image frame.
  • the processor uses the target feature to perform the three-dimensional reconstruction corresponding to the current image frame.
  • the processor may only use the target feature corresponding to the target circular region of interest to perform the three-dimensional reconstruction corresponding to the current image frame.
  • the processor can also use the target features corresponding to the target circular region of interest and the features in the non-target circular region to perform three-dimensional reconstruction corresponding to the current image frame; wherein, the non-target circular region can correspond to each target to-be-processed window.
  • a circular region of interest outside the target circular region in the circular region of interest such as a circular region of interest with less than or equal to a preset number of point features; thus, the use of such features as non-target circular regions is sparse but clear
  • the region complements the corresponding 3D reconstruction of the current image frame. This embodiment does not impose any limitation on this.
  • the embodiment of this application introduces line features in the feature extraction stage of 3D reconstruction, so that the line segment features in the 3D reconstruction scene can be better perceived, and the reconstruction efficiency of simple and regular 3D models is accelerated;
  • the region establishes the region of interest in the image, and performs feature extraction through window sliding, and performs homogenization and de-redundancy processing on the features in the 3D reconstruction in the form of a quadtree, thereby improving the efficiency and accuracy of the 3D reconstruction. , which improves the user experience.
  • FIG. 2 is a flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application.
  • the method can include:
  • Step 201 According to the preset window size, perform sliding calculation on the acquired current image frame to obtain the to-be-processed window.
  • this step is similar to step 101 and will not be repeated here.
  • Step 202 Perform point feature extraction and line feature extraction on the window to be processed, and determine the respective corresponding feature in each window to be processed; wherein, the features include point features and/or line feature constraint points.
  • the processor may perform point feature extraction and line feature extraction on the to-be-processed window after extracting the to-be-processed window from the current image frame, to obtain the point feature and line feature in the to-be-processed window. , and extract the line feature constraint point corresponding to the line feature, and use the obtained line feature constraint point and point feature as the feature corresponding to the to-be-processed window; that is, each to-be-processed window determined in this step may include multiple features, Each feature can be a point feature or a line feature constraining points.
  • the line feature is introduced in the feature extraction stage of the window to be processed, and the line feature constraint point is extracted by the line feature to establish the linear constraint relationship of the line feature, so that an anti-occlusion and high-precision
  • the method perceives the line segment features in the 3D reconstruction scene, and accelerates the reconstruction efficiency of simple and regular 3D models.
  • this embodiment does not limit the specific way of performing step 201 and this step.
  • the processor can extract a current window to be processed of the current image frame by means of window sliding, and then perform point feature extraction on the current window to be processed. and line feature extraction, determine the corresponding features in the current to-be-processed window, and then extract the next to-be-processed window; the processor may also extract all the to-be-processed windows of the current image frame, and then use this step to extract each to-be-processed window. Perform point feature extraction and line feature extraction to determine the corresponding features in each window to be processed.
  • the specific method of determining the corresponding features in each window to be processed can be set by the designer, for example, the processor can Perform point feature extraction and line feature extraction to obtain point features and line features corresponding to the current window to be processed; extract the line feature constraint points corresponding to the target number of each line feature; wherein, the current line feature corresponds to the target number of lines
  • the feature constraint point conforms to the linear constraint relationship of the current line feature, the current line feature is any line feature, and the current pending window can be any pending window.
  • the specific process of determining the corresponding features in each window to be processed can be set by the designer.
  • the processor can use ORB for the window to be processed. (Oriented FAST and Rotated BRIEF) algorithm for point feature extraction, and ED-lines (a line segment detection algorithm) algorithm for line feature extraction. If line features are extracted, LBD (Line Band Descriptor) can be used. ) algorithm establishes the descriptor of the line feature, and then performs binary conversion to obtain the binary descriptor of the line segment of the line feature (ie, the first descriptor), and then extracts three feature points (ie, the first descriptor) in the line feature as shown in Figure 4. Line feature constraint point), and establish a linear constraint relationship that conforms to the line feature.
  • ORB Oriented FAST and Rotated BRIEF
  • ED-lines a line segment detection algorithm
  • LBD Line Band Descriptor
  • the number of each corresponding line feature constraint point may be 3, that is, the above-mentioned target number may be 4.
  • three lines may be extracted from the line segment of each line feature
  • the feature constraint points are the two endpoints (p 1 and p 2 ) and the center point (p 3 ) of the line segment.
  • Step 203 Iteratively splits a circular quadtree on each target to-be-processed window to obtain a circular region of interest corresponding to each target to-be-processed window; wherein, the target to-be-processed window is a to-be-processed window with corresponding features, The number of circular ROIs corresponding to each target to-be-processed window is greater than the threshold, and the circular ROIs corresponding to each target to-be-processed window includes a first circular ROI or a first circular ROI and a second circular ROI A circular region of interest, the first circular region of interest is a circular region of interest with only one feature, and the second circular region of interest is a circular region of interest with a line feature constraint point.
  • this step may be to iteratively split the circular quadtree shown in FIG. 5 for each target pending window by the processor, and use the quadtree structure shown in Iteratively divides the dimensional image frame area, and uses a circular area (ie, a circular area of interest) to divide features, which can more accurately perceive the grayscale gradient information of feature points in the divided quadtree space.
  • a circular area ie, a circular area of interest
  • the target to-be-processed window in this step may be a to-be-processed window with corresponding features. As shown in FIG. 3 , in this step, windows without corresponding features (ie, to-be-processed windows) may be eliminated.
  • the specific method for the processor to iteratively split the circular quadtree for each target to-be-processed window in this step can be set by the designer, as long as it is ensured that after the iterative splitting, each target to-be-processed window obtained corresponds to
  • the number of circular ROIs is greater than the threshold
  • each circular ROI corresponding to the target window to be processed contains features
  • the circular ROI corresponding to each target window to be processed includes the first circular ROI area or the first circular area of interest and the second circular area of interest, which are not limited in this embodiment; wherein, the first circular area of interest is a circular area of interest with only one feature, and the first circular area of interest is a circular area of interest with only one feature.
  • a two-circle region of interest is a circular region of interest with line feature constraint points. As shown in Figure 7, this step may specifically include the following steps:
  • Step 301 Determine the root node of the current target window to be processed according to the preset window size.
  • the current target to-be-processed window in this step may be any target to-be-processed window that currently needs to perform circular quadtree iterative splitting.
  • the root node in this step may be a node in the current target window to be processed that needs to be determined for the first circular quadtree iterative split, such as the center point of a current target window to be processed (ie, the preset window size). .
  • the size of the current target window to be processed corresponding to the preset window size can be determined, and the number and position of root nodes in the current target window to be processed can be determined.
  • Step 302 Taking the root node as the center point, perform circular quadtree splitting in the current target pending window.
  • this step can be to determine the root node of the current target pending window as the center point, and perform circular quadtree splitting in the current target pending window, so as to obtain the circular sense corresponding to the four child nodes.
  • the region of interest such as the circular region of interest 1 to the circular region of interest 4 in FIG. 5 and FIG. 6 .
  • Step 303 determine whether the current circular region of interest is the second circular region of interest; if so, go to step 304 ; if not, go to step 305 .
  • the current circular region of interest in this step may be a circular region of interest containing features currently being processed; the purpose of this step may be to determine whether the current circular region of interest is the second circular region Region of interest, determine whether the current circular region of interest contains line feature constraint points, so that when the current circular region of interest contains line feature constraint points, enter step 204 and use the current circular region of interest as the current target to be processed For the circular area of interest corresponding to the window, you can no longer perform circular quadtree splitting on the current circular area of interest.
  • the specific selection of the current circular region of interest in this step can be set by the designer. For example, according to the corresponding iteration number from small to large, the circles of interest with the same number of iterations can be sequentially selected. Select a circular area of interest as the current circular area of interest.
  • Step 304 Use the current circular region of interest as the circular region of interest corresponding to the current target window to be processed.
  • each current circular region of interest is taken as the circular region of interest corresponding to the current target to-be-processed window in turn, and the number of circular regions of interest corresponding to the current target to-be-processed window is not greater than the threshold, then Continue to perform circular quadtree splitting on the circular area of interest (that is, the second circular area of interest) corresponding to the current target to-be-processed window that contains line feature constraint points, and split the obtained circular area of interest.
  • a circular region of interest is used as the current circular region of interest, and then enters step 303; for example, the second circular region of interest with the smallest number of iterations corresponding to the current target to-be-processed window or the second circular region of interest containing the largest number of features can be used.
  • a circular quadtree split is performed on the region of interest. This embodiment does not impose any limitation on this.
  • the current circular region of interest after the current circular region of interest is used as the circular region of interest corresponding to the current target to-be-processed window, it may also include a circular region of interest that has never been determined as the circular region of interest corresponding to the current target to-be-processed window Select a circular area of interest from the area of interest as the current circular area of interest, and go to step 303 . For example, if there is a circular region of interest with the same number of iterations as the current circular region of interest that is not determined to be the circular region of interest corresponding to the current target window to be processed, a circular region of interest can be selected as the current circular region of interest.
  • a circular ROI is selected as the current circular ROI among the circular ROIs of the next iteration number of iterations.
  • Step 305 determine whether the current circular region of interest is the first circular region of interest; if so, go to step 304 ; if not, go to step 306 .
  • this step may be to determine whether the current circular region of interest contains only one feature (ie point feature) by judging whether the current circular region of interest is the first circular region of interest, so as to determine whether the current circular region of interest contains only one feature (ie point feature)
  • Step 306 Determine whether the number of circular regions of interest in the current target to-be-processed window is greater than a threshold; if so, go to Step 304; if not, go to Step 307.
  • this step can be divided into the current number of circular regions of interest (that is, the number of nodes in Figure 3) and a preset threshold (that is, in Figure 3) through iterative splitting in the current target pending window. the feature point threshold), determine whether it is necessary to continue the circular quadtree iterative splitting; thus when the number of circular regions of interest in the current target to-be-processed window is greater than the threshold, enter step 304 to divide the current circular area of interest The area is used as the circular area of interest corresponding to the current target to-be-processed window, and the current circular area of interest is no longer subjected to circular quadtree splitting.
  • this embodiment it is shown as an example of determining whether to continue the iterative splitting of the circular quadtree by comparing the number of the current target to-be-processed window to the current circular region of interest and the threshold. ; It is also possible to determine whether it is necessary to continue to perform circular quadtree iterative splitting through the comparison of the number of circular regions of interest corresponding to the current target pending window and the threshold; for example, this step may be to judge the circle corresponding to the current target pending window. determine whether the number of shaped regions of interest is greater than the threshold; if yes, go to step 304 ; if not, go to step 307 .
  • the threshold for comparison with the number of circular regions of interest corresponding to the current target to-be-processed window may be smaller than the threshold in this step by 1.
  • the iterative manner of performing circular quadtree splitting on a circular area of interest is shown as an example, that is, in the current circular area of interest.
  • this step is used to directly determine whether the number of circular areas of interest in the current target pending window is greater than the threshold, so that the current target is pending processing.
  • circular quadtree splitting is directly performed on the current circular ROI.
  • the next circular area of interest with the same number of iterations as the current circular area of interest may be selected As the current circular area of interest, and return to step 303, until there is no circular area of interest with the same number of iterations as the current circular area of interest, then determine whether the number of circular areas of interest in the current target window to be processed is not. greater than the threshold; if not, all circular ROIs that are not the second circular ROI and the first circular ROI that have the same number of iterations as the current circular ROI can be rounded sequentially or simultaneously.
  • the quadtree is split, and the current circular region of interest corresponding to the next iteration number is selected to enter step 303; if so, all the iterations with the same number of iterations as the current circular region of interest may not be the second circular region of interest.
  • the area and the circular area of interest of the first circular area of interest are used as the circular area of interest corresponding to the current target to-be-processed window.
  • Step 307 perform a circular quadtree split on the current circular region of interest, and use one of the four circular regions of interest obtained from the split as the current circular region of interest, and go to step 303 .
  • this step may be when the current circular region of interest is not the first circular region of interest or the second circular region of interest, and the circular region of interest in the current target to-be-processed window
  • the number is not greater than the threshold
  • this step before using one of the four circular regions of interest obtained by splitting as the current circular region of interest, it may also include judging whether there is the same number of iterations as the current circular region of interest
  • the circular region of interest that contains features that is, the circular region of interest that is not determined as the circular region of interest corresponding to the current target to-be-processed window; Select a circular ROI as the current circular ROI from the circular ROI containing features; if it does not exist, one of the four circular ROIs obtained by splitting the current circular ROI
  • the circular ROI is used as the current circular ROI.
  • the circular quadtree iterative splitting method shown in FIG. 7 is performed by performing circular quadtree splitting on a circular area of interest (that is, the current circular area of interest) each time as an example. It is shown that the iterative splitting of the circular quadtree is performed in other ways, such as when the number of circular ROIs in the current target pending window is not greater than the threshold, and multiple ROIs that are not the first circular ROI or The circular quadtree split is performed on the circular ROI of the same iteration number of the second circular ROI. This embodiment does not impose any limitation on this.
  • the circular region of interest corresponding to each target to-be-processed window is obtained in this step, since the circular region of interest is used, some features will be retained around the region, and these features have scattered characteristics; this step can be retained.
  • the features outside the circular area of interest corresponding to the target to-be-processed window are used to prevent excessive homogenization and too few features, and can be used to correct the depth and pose information in virtual reality activities during the 3D reconstruction process.
  • Step 204 Perform feature screening on the point features in the target circular region of interest to obtain a preset number of target point features corresponding to each target circular region of interest; Circular region of interest for the number of point features.
  • this step can be to perform feature screening on the point features in the target circular region of interest, so that the number of point features in the circular region of interest corresponding to each target to-be-processed window is less than or equal to A preset number to reduce redundant point features in 3D reconstruction while preserving the line feature constraint points in the circular region of interest.
  • the specific method of performing feature screening on the point features in the target circular region of interest in this step to obtain a preset number of target point features corresponding to each target circular region of interest can be set by the designer.
  • this step can extract sampling points with a preset sampling number from around the current target circular region of interest; the gray gradient of each feature point in the current target circular region of interest is compared with Compare the gray gradients of the sampling points to obtain the binary feature quality value of each feature point in the current target circular region of interest; select from all the feature points in the current target circular region of interest according to the binary feature quality value
  • a feature point is used as the target point feature corresponding to the current target circular region of interest; wherein, the current target circular region of interest can be any target circular region of interest that contains multiple point features; as shown in Figure 8,
  • this step can extract 20 sampling points with equal perimeters on the circle outside the current target circular region of interest, and compare the gray gradient information (g i ) at these sampling points with the current target
  • the grayscale gradient information (g feature ) of each feature point in the region of interest is compared in size, and the binary feature quality value of each feature point (ie, the binary feature quality value) is obtained; for example, the current target is determined by the following formula The binary feature quality value of the current feature point in the circular region of interest;
  • g feature is the gray gradient of the current feature point
  • g i is the gray gradient of the ith sampling point
  • the feature point with the largest number of 1s in the binary feature quality value can be selected as the target point feature corresponding to the current target circular region of interest; binary features can also be selected.
  • the feature point with the largest number of 0s in the quality value is used as the target point feature corresponding to the current target circular region of interest; the target point feature corresponding to the current target circular region of interest is selected in other ways.
  • the processor can use the binary feature quality value of each feature point in the current target circular region of interest to determine the target point feature corresponding to the current target circular region of interest, this embodiment does not impose any limitation.
  • the target circular region of interest in this step may also be a circular region of interest that has more than a preset number of point features and no line feature constraint points.
  • Step 205 using the feature matching result of the target feature in the target circular region of interest and the feature matching result of the feature in the non-target circular region of interest to improve the three-dimensional reconstruction corresponding to the current image frame; wherein, the target feature includes the target point Feature or target point feature and line feature constraint point; non-target circular region of interest is a circular region of interest with less than or equal to a preset number of point features.
  • the target features in the target circular region of interest in this step may be features determined according to the target point features and line feature constraint points in the target circular region of interest, that is, the target circular region of interest may include: Multiple target point features, each target point feature or line feature constraint point in a target circular region of interest can be used as a target feature of the target circular region of interest; Correspondingly, before this step, according to each The step of determining the target feature corresponding to each target circular region of interest according to the target point features and the line feature constraint points in the target circular region of interest. There are line feature constraint points and/or point features less than or equal to a preset number in the non-target circular region of interest in this step.
  • the purpose of this step may be to use the target features in the target circular region of interest and the features in the non-target circular region of interest to perform feature matching with a preset image frame (such as the previous image frame).
  • the matching result is to perform 3D reconstruction corresponding to the current image frame for the successfully matched target features and features, and realize the geometric 3D reconstruction corresponding to the current image frame.
  • this step may include the step of performing feature matching on the target features in the target circular region of interest and the features in the non-target circular region of interest with a preset image frame to obtain a feature matching result.
  • the processor can use the binary feature quality value of each feature to be matched to perform feature matching between each feature to be matched and the previous image frame, and obtain the feature of each feature to be matched.
  • Matching results; wherein, the features to be matched include target features in the target circular region of interest and features in the non-target circular region of interest.
  • the current feature to be matched (that is, the current feature to be matched) is a point feature or a target point feature
  • the features to be matched whose feature matching results are successfully matched can be added to the three-dimensional reconstruction feature set; the features to be matched in the three-dimensional reconstruction feature set are divided into three-dimensional edge and corner reference features and three-dimensional surface reference features, and the three-dimensional edge and corner reference features are used.
  • the matching points that is, the matching point feature matching results are the features to be matched successfully matched
  • the line constraints that is, the linear constraint relationship of the corresponding line features
  • the embodiment of this application introduces line features in the feature extraction stage of 3D reconstruction, and establishes line feature constraint points through the line features, so that the line segment features in the 3D reconstruction scene can be perceived in an anti-occlusion and high-precision manner , to speed up the reconstruction efficiency of simple and regular 3D models; and the application uses a circular area to establish a region of interest in the image, and performs feature extraction by window sliding, and homogenizes the features in the 3D reconstruction by a quadtree. and de-redundancy processing, thereby improving the efficiency and accuracy of 3D reconstruction and improving user experience
  • FIG. 9 is a structural block diagram of a three-dimensional reconstruction apparatus provided by an embodiment of the present application.
  • the apparatus may include:
  • the sliding calculation module 10 is configured to perform sliding calculation on the acquired current image frame according to the preset window size to obtain the to-be-processed window;
  • the feature extraction module 20 is used to perform point feature extraction and line feature extraction on the window to be processed, and determine the respective corresponding features in each window to be processed;
  • the iterative splitting module 30 is used for performing circular quadtree iterative splitting on each target to-be-processed window to obtain a circular region of interest corresponding to each target to-be-processed window;
  • the feature screening module 40 is configured to perform feature screening on the features in the target circular region of interest to obtain target features corresponding to each target circular region of interest;
  • the three-dimensional reconstruction module 50 is configured to perform three-dimensional reconstruction corresponding to the current image frame by using the target feature.
  • each to-be-processed window may include point features and/or line feature constraint points.
  • the target to-be-processed window may be a to-be-processed window with corresponding features, the number of circular regions of interest corresponding to each target to-be-processed window may be greater than a threshold, and each target to-be-processed window corresponds to a circular area of interest.
  • the area may include a first circular area of interest or a first circular area of interest and a second circular area of interest, the first circular area of interest may be a circular area of interest with only one feature, and the second circular area of interest may be a circular area of interest with only one feature.
  • the shaped region of interest may be a circular region of interest with line feature constraint points.
  • the feature screening module 40 may include:
  • the point feature screening sub-module is used to perform feature screening on the point features in the target circular region of interest, and obtain a preset number of target point features in each target circular region of interest; wherein, the target circular region of interest is There are circular regions of interest with greater than a preset number of point features;
  • the determination sub-module is used to determine the corresponding target features of each target circular area of interest according to the target point features and line feature constraint points in each target circular area of interest; wherein, each target circular area of interest is of interest
  • the respective target features corresponding to the regions include target point features, or target point features and line feature constraint points.
  • the feature extraction module 20 may include:
  • the first extraction sub-module is used for extracting point features and line features of the current window to be processed, to obtain point features and line features corresponding to the current window to be processed;
  • the second extraction sub-module is used to extract the corresponding target number of line feature constraint points for each line feature; wherein, the target number of line feature constraint points corresponding to the current line feature conform to the linear constraint relationship of the current line feature, and the current line
  • the feature is any line feature.
  • the iterative splitting module 30 may include:
  • the root node determination submodule is used to determine the root node of the current target to-be-processed window according to the preset window size
  • the split sub-module is used to split the circular quadtree in the current target pending window with the root node as the center point;
  • the first judgment sub-module is used to judge whether the current circular region of interest is the second circular region of interest; if it is the second circular region of interest, the current circular region of interest is used as the current target to be processed corresponding to the window the circular region of interest;
  • the second judging submodule is used to judge whether the current circular ROI is the first circular ROI if it is not the second circular ROI; if it is the first circular ROI, then the current circular ROI is The shape region of interest is taken as the circle region of interest corresponding to the current target to-be-processed window; wherein, the first circle region of interest is a circle region of interest with only one point feature.
  • the iterative splitting module 30 may further include:
  • the third judging sub-module is used to judge whether the number of circular ROIs in the current target to-be-processed window is greater than the threshold if it is not the first circular ROI; if it is greater than the threshold, the current circular ROI is The area is used as the circular area of interest corresponding to the current target pending window;
  • the iterative submodule is used to split the current circular area of interest by a circular quadtree if it is not greater than the threshold, and use one of the four circular areas of interest obtained from the split as the current circle A region of interest is formed, and a start signal is sent to the first judgment sub-module.
  • the feature screening module 40 may include:
  • a sampling sub-module used to extract sampling points with a preset sampling number from the surroundings of the current target circular region of interest
  • the comparison sub-module is used to compare the gray gradient of each feature point in the current target circular region of interest with the gray gradient of the sampling point to obtain the binary value of each feature point in the current target circular region of interest. feature quality value;
  • the screening sub-module is used to select a feature point from all the feature points in the current target circular region of interest as the target point feature corresponding to the current target circular region of interest according to the binary feature quality value.
  • the comparison sub-module can be specifically used to pass Determining the binary value of the current mass characteristic feature point circular current target region of interest; wherein, g feature shades of gray of the current feature point, g i is the gray levels of the i-th sampling point.
  • the three-dimensional reconstruction module 50 can be specifically used to improve the three-dimensional reconstruction corresponding to the current image frame by using the feature matching result of the target feature and the feature matching result of the feature in the non-target circular region of interest; wherein, the non-target circle
  • the shaped region of interest is a circular region of interest other than the target circular region of interest in the circular region of interest corresponding to each target to-be-processed window.
  • the three-dimensional reconstruction module 50 may include:
  • the feature matching submodule is used to perform feature matching between each feature to be matched and the previous image frame by using the binary feature quality value of each feature to be matched, and obtain the feature matching result of each feature to be matched; wherein, the feature to be matched is Including target features in the target circular region of interest and features in the non-target circular region of interest;
  • the allocation sub-module is used to add the to-be-matched features whose feature matching result is successful matching into the 3D reconstruction feature set;
  • the division and reconstruction sub-module is used to divide the to-be-matched features in the 3D reconstruction feature set into 3D edge and corner reference features and 3D surface reference features, and use the 3D edge and corner reference features and the 3D surface reference features to perform 3D reconstruction corresponding to the current image frame; wherein,
  • the three-dimensional edge reference feature is a feature to be matched that satisfies the linear constraint relationship of the corresponding line feature.
  • the feature matching sub-module may be specifically configured to use the target descriptor corresponding to the current feature to be matched to perform feature matching between the current feature to be matched and the previous image frame; wherein,
  • the target descriptor is the sum of the first descriptor and the second descriptor, the first descriptor is the binary descriptor calculated by the LBD algorithm for the line feature corresponding to the current feature to be matched, and the second descriptor is the current feature to be matched.
  • Binary feature quality value is the target descriptor corresponding to the current feature to be matched.
  • the embodiment of this application introduces line features in the feature extraction stage of 3D reconstruction, so that the line segment features in the 3D reconstruction scene can be better perceived, and the reconstruction efficiency of simple and regular 3D models is accelerated;
  • the region establishes the region of interest in the image, and performs feature extraction through window sliding, and performs homogenization and de-redundancy processing on the features in the 3D reconstruction in the form of a quadtree, thereby improving the efficiency and accuracy of the 3D reconstruction. , which improves the user experience.
  • FIG. 10 is a schematic structural diagram of a three-dimensional reconstruction device provided by an embodiment of the present application.
  • the device 1 may include:
  • the memory 11 is used to store a computer program; the processor 12 is used to implement the steps of the three-dimensional reconstruction method provided by the above embodiments when the computer program is executed.
  • the device 1 may specifically be a VR device, an AR device, an MR device, or other devices that need to perform three-dimensional reconstruction, and the device 1 may include a memory 11 , a processor 12 and a bus 13 .
  • the memory 11 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like.
  • the memory 11 may be an internal storage unit of the device 1 in some embodiments. In other embodiments, the memory 11 may also be an external storage device of the device 1, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure) device equipped on an AR device, a VR device, or an MR device. Digital, SD) card, flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the device 1 and an external storage device.
  • the memory 11 can not only be used to store application software installed in the device 1 and various types of data, such as code of a program executing the three-dimensional reconstruction method, etc., but also can be used to temporarily store data that has been output or will be output.
  • the processor 12 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor or other data processing chip in some embodiments, for running the program code or processing stored in the memory 11 Data, such as the code of a program that executes a three-dimensional reconstruction method, etc.
  • CPU Central Processing Unit
  • controller microcontroller
  • microprocessor or other data processing chip in some embodiments, for running the program code or processing stored in the memory 11 Data, such as the code of a program that executes a three-dimensional reconstruction method, etc.
  • the bus 13 may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (EISA for short) bus or the like.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 10, but it does not mean that there is only one bus or one type of bus.
  • the device may also include a network interface 14, and the network interface 14 may optionally include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used between the device 1 and other electronic devices. establish a communication connection.
  • a network interface 14 may optionally include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used between the device 1 and other electronic devices. establish a communication connection.
  • the device 1 may further include a user interface 15, the user interface 15 may include a display, an input unit such as a button, and the optional user interface 15 may also include a standard wired interface and a wireless interface.
  • the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like.
  • the display may also be appropriately called a display screen or a display unit, for displaying information processed in the device 1 and for displaying a visual user interface.
  • FIG. 10 only shows the device 1 with components 11-15. Those skilled in the art can understand that the structure shown in FIG. 10 does not constitute a limitation on the device 1, and may include fewer or more components than those shown in the drawings. components, or a combination of certain components, or a different arrangement of components.
  • an embodiment of the present application further discloses a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method provided by the above-mentioned embodiments are implemented. .
  • the storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other various storage media that can store program codes medium.
  • U disk mobile hard disk
  • read-only memory Read-Only Memory
  • RAM random access memory
  • magnetic disk or optical disk and other various storage media that can store program codes medium.
  • a software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.
  • RAM random access memory
  • ROM read only memory
  • electrically programmable ROM electrically erasable programmable ROM
  • registers hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Multimedia (AREA)
  • Architecture (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Image Analysis (AREA)

Abstract

一种三维重建方法，包括：对获取的当前图像帧进行滑动计算，得到待处理窗口；对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征；对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域；对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征；利用所述目标特征，进行当前图像帧对应的三维重建；在三维重建的特征提取阶段引入线特征，从而更好的感知三维重建场景中的线段特征，加速简单规则三维模型的重建效率；且通过圆形四叉树迭代分裂的使用，提高三维重建的效率和精度，提升用户体验。

Description

一种三维重建方法、装置、设备及计算机可读存储介质
本申请要求于2020年07月17日提交中国专利局、申请号为202010692404.7、发明名称为“一种三维重建方法、装置、设备及计算机可读存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及头戴显示设备领域，特别涉及一种三维重建方法、装置、设备及计算机可读存储介质。
背景技术
目前高端类型的VR(Virtual Reality，虚拟现实)设备、AR(Augmented Reality，增强现实)设备和MR(Mediated Reality，混合现实)设备等头戴显示设备在使用过程中，往往会通过视觉SLAM(Simultaneous Localization And Mapping，同步定位与建图)技术对使用场景进行三维重建。
现有技术中，在大多数应用场景中，头戴显示设备往往只需要重建出复杂场景的简单几何构造来满足用户的定位及虚实交互需求即可，但由于真实场景中存在大量的特征信息，因此会导致三维重建过程存在因冗余特征信息过多和特征质量差而导致的重建耗时和重建效果差的缺点，严重制约了头戴显示设备的应用。
因此，如何能够提高头戴显示设备对规则物体的三维重建的效率和精度，提升用户体验，是现今急需解决的问题。
发明内容
本申请的目的是提供一种三维重建方法、装置、设备及计算机可读存储介质，以提高头戴显示设备对规则物体的三维重建的效率和精度，提升用户体验。
为解决上述技术问题，本申请提供一种三维重建方法，包括：
根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗 口；
对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征；
对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域；
对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征；
利用所述目标特征，进行当前图像帧对应的三维重建。
可选的，每个所述待处理窗口中各自对应的特征包括点特征和/或线特征约束点。
可选的，所述目标待处理窗口为存在对应的特征的待处理窗口，每个所述目标待处理窗口对应的圆形感兴趣区域的数量大于阈值，每个所述目标待处理窗口对应的圆形感兴趣区域包括第一圆形感兴趣区域或所述第一圆形感兴趣区域和第二圆形感兴趣区域，所述第一圆形感兴趣区域为只存在一个所述特征的圆形感兴趣区域，所述第二圆形感兴趣区域为存在所述线特征约束点的圆形感兴趣区域。
可选的，所述对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征，包括：
对所述目标圆形感兴趣区域中的所述点特征进行特征筛选，得到每个所述目标圆形感兴趣区域中预设数量的目标点特征；其中，所述目标圆形感兴趣区域为存在大于所述预设数量的所述点特征的圆形感兴趣区域；
根据每个所述目标圆形感兴趣区域中的所述目标点特征和所述线特征约束点，确定每个所述目标圆形感兴趣区域各自对应的目标特征；其中，每个所述目标圆形感兴趣区域各自对应的目标特征包括所述目标点特征，或所述目标点特征与所述线特征约束点。
可选的，所述对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征，包括：
对当前待处理窗口进行点特征提取和线特征提取，得到当前待处理窗口对应的点特征和线特征；
提取每个所述线特征各自的对应的目标数量的线特征约束点；其中，当 前线特征对应的所述目标数量的线特征约束点符合当前线特征的线性约束关系，当前线特征为任一所述线特征。
可选的，所述对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域，包括：
根据所述预设窗口尺寸，确定当前目标待处理窗口的根节点；
以所述根节点为中心点，在当前目标待处理窗口进行圆形四叉树分裂；
判断当前圆形感兴趣区域是否为所述第二圆形感兴趣区域；
若是，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
若否，则判断当前圆形感兴趣区域是否为所述第一圆形感兴趣区域。
可选的，若当前圆形感兴趣区域为所述第一圆形感兴趣区域，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
若当前圆形感兴趣区域不为所述第一圆形感兴趣区域，则判断当前目标待处理窗口中的圆形感兴趣区域的数量是否大于所述阈值；
若大于所述阈值，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
若不大于所述阈值，则对当前圆形感兴趣区域进行圆形四叉树分裂，并将分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域，执行所述判断当前圆形感兴趣区域是否为所述第二圆形感兴趣区域的步骤。
可选的，所述预设数量为1时，所述对目标圆形感兴趣区域中的点特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的预设数量的目标点特征，包括：
从当前目标圆形感兴趣区域的周围提取预设采样数量的采样点；
将当前目标圆形感兴趣区域中的每个特征点的灰度梯度与所述采样点的灰度梯度进行对比，得到当前目标圆形感兴趣区域中的每个特征点的二进制特征质量值；
根据所述二进制特征质量值，从当前目标圆形感兴趣区域中的全部特征点中选择一个特征点作为当前目标圆形感兴趣区域对应的目标点特征。
可选的，所述预设采样数量为20时，所述将当前目标圆形感兴趣区域中 的每个特征点的灰度梯度与所述采样点的灰度梯度进行对比，得到当前目标圆形感兴趣区域中的每个特征点的二进制特征质量值，包括：
通过
Figure PCTCN2020127362-appb-000001
确定当前目标圆形感兴趣区域中当前特征点的二进制特征质量值；其中，g feature为当前特征点的灰度梯度，g i为第i个采样点的灰度梯度。
可选的，所述利用所述目标特征，进行当前图像帧对应的三维重建，包括：
利用所述目标特征的特征匹配结果和非目标圆形感兴趣区域中的特征的特征匹配结果，完善当前图像帧对应的三维重建；其中，所述非目标圆形感兴趣区域为每个所述目标待处理窗口对应的圆形感兴趣区域中所述目标圆形感兴趣区域以外的圆形感兴趣区域。
可选的，所述利用所述目标特征，进行当前图像帧对应的三维重建，包括：
利用每个待匹配特征的二进制特征质量值，将每个所述待匹配特征与前一图像帧进行特征匹配，得到每个所述待匹配特征的特征匹配结果；其中，所述待匹配特征包括所述目标特征；
将所述特征匹配结果为匹配成功的所述待匹配特征加入三维重建特征集；
将所述三维重建特征集中的所述待匹配特征划分为三维棱角参考特征和三维面参考特征，并利用所述三维棱角参考特征和所述三维面参考特征进行当前图像帧对应的三维重建；其中，所述三维棱角参考特征为满足对应的线特征的线性约束关系的所述待匹配特征。
可选的，当前待匹配特征为线特征约束点时，所述利用每个待匹配特征的二进制特征质量值，将每个所述待匹配特征与前一图像帧进行特征匹配，得到每个所述待匹配特征的特征匹配结果，包括：
利用当前待匹配特征对应的目标描述子，将当前待匹配特征与前一图像帧进行特征匹配；其中，所述目标描述子为第一描述子与第二描述子之和，所述第一描述子为当前待匹配特征所对应的线特征通过LBD算法计算得到 的二进制描述子，所述第二描述子为当前待匹配特征的二进制特征质量值。
本申请还提供了一种三维重建装置，包括：
滑动计算模块，用于根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口；
特征提取模块，用于对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征；
迭代分裂模块，用于对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域；
特征筛选模块，用于对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征；
三维重建模块，用于利用所述目标特征，进行当前图像帧对应的三维重建。
本申请还提供了一种三维重建设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述计算机程序时实现如上述所述的三维重建方法的步骤。
本申请还提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如上述所述的三维重建方法的步骤。
本申请所提供的一种三维重建方法，包括：根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口；对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征；对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域；对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征；利用所述目标特征，进行当前图像帧对应的三维重建；
可见，本申请在三维重建的特征提取阶段引入线特征，从而能够更好的感知三维重建场景中的线段特征，加速简单规则三维模型的重建效率；并且本申请采用圆形区域建立图像中的感兴趣区域，并通过窗口滑动的方式进行特征提取，以四叉树的方式对三维重建中的特征进行均匀化和去冗余化处理， 从而提高了三维重建的效率和精度，提升了用户体验。此外，本申请还提供了一种三维重建装置、设备及计算机可读存储介质，同样具有上述有益效果。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一部分附图，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。
图1为本申请实施例所提供的一种三维重建方法的流程图；
图2为本申请实施例所提供的另一种三维重建方法的流程图；
图3为本申请实施例所提供的另一种三维重建方法的流程示意图；
图4为本申请实施例所提供的一种线特征的线特征约束点的展示图；
图5为本申请实施例所提供的一种圆形四叉树迭代分裂的展示图；
图6为本申请实施例所提供的一种圆形四叉树迭代分裂的结构示意图；
图7为本申请实施例所提供的一种圆形四叉树迭代分裂的流程图；
图8为本申请实施例所提供的一种圆形感兴趣区域的采样示意图；
图9为本申请实施例所提供的一种三维重建装置的结构框图；
图10为本申请实施例所提供的一种三维重建设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
请参考图1，图1为本申请实施例所提供的一种三维重建方法的流程图。该方法可以包括：
步骤101：根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口。
可以理解的是，本步骤的目的可以为三维重建设备(如VR设备、AR设备、MR设备等)中的处理器在当前的图像帧上提取预设窗口尺寸的窗口(即 待处理窗口)。
具体的，本步骤中得到的待处理窗口的窗口数量可以由当前图像帧的尺寸和预设窗口尺寸决定，如预设窗口尺寸为预先设定的窗口长度时，具体关系可以如下：
Figure PCTCN2020127362-appb-000002
上式中，N可以为待处理窗口的窗口数量；L可以为当前图像帧的长度；l可以为当前图像帧的裁剪区长度，通过l的设置可以在特征提取过程中避免边缘畸变带来的特征提取误差；l'可以为预先设定的窗口长度。
步骤102：对待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征。
可以理解的是，本步骤可以为处理器在从当前图像帧中提取到待处理窗口后，对该待处理窗口进行点特征提取和线特征提取，提取得到该待处理窗口中的点特征和线特征所对应的数据(即特征)；即本步骤中确定的每个待处理窗口中可以包括多个特征。
具体的，对于本步骤中提取得到的每个待处理窗口中各自对应的特征的具体内容，可以有设计人员根据使用场景和用户需求自行设置，如仅对待处理窗口进行点特征提取和线特征提取时，每个待处理窗口中各自对应的特征可以包括点特征和/或线特征，即每个特征可以为点特征或线特征。如在一个对待处理窗口进行点特征提取和线特征提取后，还可以提取得到的线特征对应的线特征约束点，将得到的线特征约束点和点特征作为该待处理窗口对应的特征，即每个待处理窗口中各自对应的特征可以包括点特征和/或线特征对应的线特征约束点，每个特征可以为点特征或线特征约束点，从而通过提取线特征对应的线特征约束点，能够以一种抗遮挡和高精度的方式感知三维重建场景中的线段特征，进一步加速简单规则三维模型的重建效率。本实施例对此不作任何限制。
具体的，本实施例并不限定步骤101与本步骤的具体进行方式，如处理器可以通过窗口滑动的方式，提取当前图像帧的一个当前待处理窗口，再对当前待处理窗口进行点特征提取和线特征提取，确定当前待处理窗口中对应的特征，再提取下一待处理窗口；处理器也可以处理器先提取当前图像帧的 全部待处理窗口，再通过本步骤对每个待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征。
步骤103：对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个目标待处理窗口对应的圆形感兴趣区域。
其中，本步骤中的目标待处理窗口可以为需要进行圆形四叉树迭代分裂的待处理窗口。对于目标待处理窗口的具体选择，可以有设计人员根据实用场景和用户需求自行设置，如目标待处理窗口可以为全部待处理窗口，即本步骤可以对当前图像帧对应的每个待处理窗口进行圆形四叉树迭代分裂；目标待处理窗口也可以为存在对应的特征的待处理窗口，即本步骤可以对通过步骤102提取到对应特征的待处理窗口进行圆形四叉树迭代分裂；目标待处理窗口还可以为存在特征数量大于特征阈值和/或存在所需特征(如线特征约束点)的待处理窗口。本实施例对此不做任何限制。
可以理解的是，本步骤的目的可以为通过处理器对每个目标待处理窗口进行如图5所示的圆形四叉树迭代分裂，利用如图6所示的四叉树结构来对二维图像帧区域进行迭代划分，同时利用圆形区域(即圆形感兴趣区域)来划分特征，可以更准确的感知特征在所划分的四叉树空间中的灰度梯度信息。
具体的，对于本步骤中得到的每个目标待处理窗口对应的圆形感兴趣区域的具体内容，即对每个目标待处理窗口进行圆形四叉树迭代分裂的具体方式，可以由设计人员根据实用场景和用户需求自行设置，如每个目标待处理窗口对应的圆形感兴趣区域的数量可以大于阈值，目标待处理窗口对应的每个圆形感兴趣区域可以均包含特征，且每个目标待处理窗口对应的圆形感兴趣区域中可以包括第一圆形感兴趣区域或第一圆形感兴趣区域和第二圆形感兴趣区域；其中，第一圆形感兴趣区域可以为特征数量小于或等于预设数量的圆形感兴趣区域，如预设数量为1时，第一圆形感兴趣区域可以为只存在一个特征的圆形感兴趣区域，第二圆形感兴趣区域可以为存在线特征对应的特征(如线特征约束点)的圆形感兴趣区域。本实施例对此不作任何限制。
步骤104：对目标圆形感兴趣区域中的特征进行特征筛选，得到每个目标圆形感兴趣区域对应的目标特征。
其中，本步骤中的目标圆形感兴趣区域可以为需要进行特征筛选的圆形感兴趣区域。对于目标圆形感兴趣区域的具体选择，可以有设计人员根据实 用场景和用户需求自行设置，如目标圆形感兴趣区域可以为存在大于预设数量的点特征的圆形感兴趣区域；也就是说，本步骤的特征筛选可以仅对目标圆形感兴趣区域中的点特征进行特征筛选，从而得到每个目标圆形感兴趣区域中筛选后的预设数量的点特征(即目标点特征)；例如本步骤可以包括对目标圆形感兴趣区域中的点特征进行特征筛选，得到每个目标圆形感兴趣区域中预设数量的目标点特征；根据每个目标圆形感兴趣区域中的目标点特征和线特征约束点，确定每个目标圆形感兴趣区域各自对应的目标特征；其中，每个目标圆形感兴趣区域各自对应的目标特征包括目标点特征，或目标点特征与线特征约束点。目标圆形感兴趣区域也可以为包含线特征约束点和点特征的圆形感兴趣区域以及包含大于预设数量的点特征且不包含线特征约束点的圆形感兴趣区域；也就是说，本步骤的特征筛选可以筛除包含线特征约束点的目标圆形感兴趣区域中的点特征，以及对包含大于预设数量的点特征的目标圆形感兴趣区域中的点特征进行特征筛选，得到目标圆形感兴趣区域中筛选后的预设数量的点特征，使得一个目标圆形感兴趣区域对应的目标特征可以为存在的线特征或预设数量的点特征。本实施例对此不作任何限制。
步骤105：利用目标特征，进行当前图像帧对应的三维重建。
可以理解的是，本步骤的目的可以为利用得到的每个目标圆形感兴趣区域对应的目标特征，进行当前图像帧对应的三维重建。
具体的，对于本步骤中处理器利用目标特征，进行当前图像帧对应的三维重建的具体方式，如处理器可以仅利用目标圆形感兴趣区域对应的目标特征，进行当前图像帧对应的三维重建。处理器也可以利用目标圆形感兴趣区域对应的目标特征和非目标圆形区域中的特征，进行当前图像帧对应的三维重建；其中，非目标圆形区域可以为每个目标待处理窗口对应的圆形感兴趣区域中目标圆形区域外的圆形感兴趣区域，如存在小于或等于预设数量的点特征的圆形感兴趣区域；从而利用非目标圆形区域这样的特征稀疏但明确的区域补充完善当前图像帧对应的三维重建。本实施例对此不作任何限制。
本实施例中，本申请实施例通过在三维重建的特征提取阶段引入线特征，从而能够更好的感知三维重建场景中的线段特征，加速简单规则三维模型的重建效率；并且本申请采用圆形区域建立图像中的感兴趣区域，并通过窗口滑动的方式进行特征提取，以四叉树的方式对三维重建中的特征进行均匀化 和去冗余化处理，从而提高了三维重建的效率和精度，提升了用户体验。
请参考图2，图2为本申请实施例所提供的一种三维重建方法的流程图。该方法可以包括：
步骤201：根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口。
其中，本步骤与步骤101相似，在此不再赘述。
步骤202：对待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征；其中，特征包括点特征和/或线特征约束点。
可以理解的是，本步骤可以为处理器在从当前图像帧中提取到待处理窗口后，对该待处理窗口进行点特征提取和线特征提取，得到该待处理窗口中的点特征和线特征，并提取得到线特征对应的线特征约束点，将得到的线特征约束点和点特征作为该待处理窗口对应的特征；即本步骤中确定的每个待处理窗口中可以包括多个特征，每个特征可以为点特征或线特征约束点。
也就是说，本实施例中通过在待处理窗口的特征提取阶段引入线特征，并通过线特征提取线特征约束点，建立线特征的线性约束关系，从而能够以一种抗遮挡和高精度的方式感知三维重建场景中的线段特征，加速简单规则三维模型的重建效率。
具体的，本实施例并不限定步骤201与本步骤的具体进行方式，如处理器可以通过窗口滑动的方式，提取当前图像帧的一个当前待处理窗口，再对当前待处理窗口进行点特征提取和线特征提取，确定当前待处理窗口中对应的特征，再提取下一待处理窗口；处理器也可以处理器先提取当前图像帧的全部待处理窗口，再通过本步骤对每个待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征。
对应的，对于本步骤中对待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征的具体方式，可以由设计人员自行设置，如处理器可以对当前待处理窗口进行点特征提取和线特征提取，得到当前待处理窗口对应的点特征和线特征；提取每个线特征各自的对应的目标数量的线特征约束点；其中，当前线特征对应的目标数量的线特征约束点符合当前线特征的线性约束关系，当前线特征为任一线特征，当前待处理窗口可以为 任一待处理窗口。对应的，若对当前待处理窗口进行点特征提取和线特征提取，未得到当前待处理窗口对应的线特征，则可以直接结束对当前待处理窗口的特征提取，将得到的当前待处理窗口对应的点特征作为当前待处理窗口中对应的特征。本实施例对此不做任何限制。
具体的，对于本步骤中对待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征的具体过程，可以由设计人员自行设置，如处理器可以对待处理窗口利用ORB(Oriented FAST and Rotated BRIEF)算法进行点特征提取，并利用ED-lines(一种线段检测算法)算法进行线特征提取，若提取到线特征，则可以利用LBD(Line Band Descriptor，线带描述子)算法建立线特征的描述子，然后进行二进制转换，得到该线特征的线段的二进制描述子(即第一描述子)，之后按照图4所示，提取出线特征中的三个特征点(即线特征约束点)，并建立起符合该线特征的线性约束关系。
也就是说，如图4所示，每个各自的对应的线特征约束点的数量可以为3，即上述目标数量可以为4，本步骤中可以从每个线特征的线段中提取三个线特征约束点，即线段的两个端点(p 1和p 2)和中心点(p 3)。
步骤203：对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个目标待处理窗口对应的圆形感兴趣区域；其中，目标待处理窗口为存在对应的特征的待处理窗口，每个目标待处理窗口对应的圆形感兴趣区域的数量大于阈值，每个目标待处理窗口对应的圆形感兴趣区域包括第一圆形感兴趣区域或第一圆形感兴趣区域和第二圆形感兴趣区域，第一圆形感兴趣区域为只存在一个特征的圆形感兴趣区域，第二圆形感兴趣区域为存在线特征约束点的圆形感兴趣区域。
可以理解的是，本步骤的目的可以为通过处理器对每个目标待处理窗口进行如图5所示的圆形四叉树迭代分裂，利用如图6所示的四叉树结构来对二维图像帧区域进行迭代划分，同时利用圆形区域(即圆形感兴趣区域)来划分特征，可以更准确的感知特征点在所划分的四叉树空间中的灰度梯度信息。
其中，本步骤中的目标待处理窗口可以为存在对应的特征的待处理窗口，如图3所示，本步骤中可以剔除掉不含对应的特征的窗口(即待处理窗口)。
具体的，对于本步骤中处理器对每个目标待处理窗口进行圆形四叉树迭 代分裂的具体方式，可以由设计人员自行设置，只要保证迭代分裂后，得到的每个目标待处理窗口对应的圆形感兴趣区域的数量大于阈值，目标待处理窗口对应的每个圆形感兴趣区域均包含特征，且每个目标待处理窗口对应的圆形感兴趣区域中包括第一圆形感兴趣区域或第一圆形感兴趣区域和第二圆形感兴趣区域，本实施例对此不做任何限制；其中，第一圆形感兴趣区域为只存在一个特征的圆形感兴趣区域，第二圆形感兴趣区域为存在线特征约束点的圆形感兴趣区域。如图7所示，本步骤可以具体包括如下步骤：
步骤301：根据预设窗口尺寸，确定当前目标待处理窗口的根节点。
其中，本步骤中的当前目标待处理窗口可以为任意一个当前需要进行圆形四叉树迭代分裂的目标待处理窗口。本步骤中的根节点可以为当前目标待处理窗口中用于进行第一次圆形四叉树迭代分裂所需要确定的节点，如一个当前目标待处理窗口(即预设窗口尺寸)的中心点。
具体的，如图3所示，本步骤可以确定由预设窗口尺寸对应的当前目标待处理窗口的尺寸，确定当前目标待处理窗口中根节点的数量和位置。
步骤302：以根节点为中心点，在当前目标待处理窗口进行圆形四叉树分裂。
可以理解的是，本步骤的目的可以为以确定的当前目标待处理窗口的根节点为中心点，在当前目标待处理窗口进行圆形四叉树分裂，从而得到四个子节点对应的圆形感兴趣区域，如图5和图6中的圆形感兴趣区域1至圆形感兴趣区域4。
步骤303：判断当前圆形感兴趣区域是否为第二圆形感兴趣区域；若是，则进入步骤304；若否，则进入步骤305。
其中，本步骤中的当前圆形感兴趣区域可以为在一个当前进行处理的包含有特征的圆形感兴趣区域；本步骤的目的可以为通过判断当前圆形感兴趣区域是否为第二圆形感兴趣区域，确定当前圆形感兴趣区域中是否含有线特征约束点，从而在当前圆形感兴趣区域中含有线特征约束点时，进入步骤204将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域，可以不再对当前圆形感兴趣区域进行圆形四叉树分裂。
具体的，对于本步骤中当前圆形感兴趣区域的具体选择，可以由设计人员自行设置，如可以按照对应的迭代次数由小到大的顺序，依次从迭代次数 相同的多个圆形感兴趣区域中选择一个圆形感兴趣区域作为当前圆形感兴趣区域。
步骤304：将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域。
可以理解的是，本步骤通过将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域，可以避免对当前圆形感兴趣区域再次进行圆形四叉树分裂。
对应的，若依次将每个当前圆形感兴趣区域均作为当前目标待处理窗口对应的圆形感兴趣区域后，当前目标待处理窗口对应的圆形感兴趣区域的数量不大于阈值，则可以对当前目标待处理窗口对应的含有线特征约束点的圆形感兴趣区域(即第二圆形感兴趣区域)继续进行圆形四叉树分裂，并将分裂得到的圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域，进入步骤303；例如可以对当前目标待处理窗口对应的迭代次数最小的第二圆形感兴趣区域或包含特征数量最多的第二圆形感兴趣区域进行圆形四叉树分裂。本实施例对此不做任何限制。
具体的，本步骤中将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域之后，还可以包括从未确定为当前目标待处理窗口对应的圆形感兴趣区域的圆形感兴趣区域中选择一个圆形感兴趣区域作为当前圆形感兴趣区域，并进入步骤303。例如若存在未确定为当前目标待处理窗口对应的圆形感兴趣区域的与当前圆形感兴趣区域的迭代次数相同的圆形感兴趣区域，可以从中选择一个圆形感兴趣区域作为当前圆形感兴趣区域；若不存在未确定为当前目标待处理窗口对应的圆形感兴趣区域的与当前圆形感兴趣区域的迭代次数相同的圆形感兴趣区域，可以从当前圆形感兴趣区域的迭代次数的下一迭代次数的圆形感兴趣区域中选择一个圆形感兴趣区域作为当前圆形感兴趣区域。
步骤305：判断当前圆形感兴趣区域是否为第一圆形感兴趣区域；若是，则进入步骤304；若否，则进入步骤306。
其中，本步骤的目的可以为通过判断当前圆形感兴趣区域是否为第一圆形感兴趣区域，确定当前圆形感兴趣区域中是否仅含有一个特征(即点特征)，从而在当前圆形感兴趣区域中仅含有一个特征时，进入步骤304将当前圆形 感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域，不再对当前圆形感兴趣区域进行圆形四叉树分裂。
步骤306：判断当前目标待处理窗口中的圆形感兴趣区域的数量是否大于阈值；若是，则进入步骤304；若否，则进入步骤307。
可以理解的是，本步骤的目的可以为通过当前目标待处理窗口中的迭代分裂到当前的圆形感兴趣区域的数量(即图3中的节点数量)与预先设置的阈值(即图3中的特征点阈值)的比较，确定是否需要继续进行圆形四叉树迭代分裂；从而在当前目标待处理窗口中的圆形感兴趣区域的数量大于阈值时，进入步骤304将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域，不再对当前圆形感兴趣区域进行圆形四叉树分裂。
具体的，本实施例是以通过当前目标待处理窗口中的迭代分裂到当前的圆形感兴趣区域的数量与阈值的比较，确定是否需要继续进行圆形四叉树迭代分裂为例进行的展示；也可以通过当前目标待处理窗口对应的圆形感兴趣区域的数量与阈值的比较，确定是否需要继续进行圆形四叉树迭代分裂；例如本步骤可以为判断当前目标待处理窗口对应的圆形感兴趣区域的数量是否大于阈值；若是，则进入步骤304；若否，则进入步骤307。相应的，与当前目标待处理窗口对应的圆形感兴趣区域的数量进行比较的阈值可以比本步骤中的阈值小1。
需要说明的是，本实施例是以每次对一个圆形感兴趣区域(即当前圆形感兴趣区域)进行圆形四叉树分裂的迭代方式为例进行的展示，即在当前圆形感兴趣区域不为第二圆形感兴趣区域和第一圆形感兴趣区域时，通过本步骤直接判断当前目标待处理窗口中的圆形感兴趣区域的数量是否大于阈值，从而在当前目标待处理窗口中的圆形感兴趣区域的数量不大于阈值时，直接对当前圆形感兴趣区域进行圆形四叉树分裂。本步骤也可以在当前圆形感兴趣区域不为第二圆形感兴趣区域且第一圆形感兴趣区域时，选择下一个与当前圆形感兴趣区域的迭代次数相同的圆形感兴趣区域作为当前圆形感兴趣区域，并返回步骤303，直至没有与当前圆形感兴趣区域的迭代次数相同的圆形感兴趣区域后再判断当前目标待处理窗口中的圆形感兴趣区域的数量是否大于阈值；若否，则可以依次或同时对与当前圆形感兴趣区域的迭代次数相同的全部不为第二圆形感兴趣区域和第一圆形感兴趣区域的圆形感兴趣区域进 行圆形四叉树分裂，并选择下一迭代次数对应的当前圆形感兴趣区域进入步骤303；若是，则可以将与当前圆形感兴趣区域的迭代次数相同的全部不为第二圆形感兴趣区域和第一圆形感兴趣区域的圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域。
步骤307：对当前圆形感兴趣区域进行圆形四叉树分裂，并将分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域，进入步骤303。
可以理解的是，本步骤的目的可以为在当前圆形感兴趣区域不为第一圆形感兴趣区域或第二圆形感兴趣区域，且当前目标待处理窗口中的圆形感兴趣区域的数量不大于阈值时，通过直接对当前圆形感兴趣区域进行圆形四叉树分裂，以增加当前目标待处理窗口中的圆形感兴趣区域的数量，通过将分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域并进入步骤303，继续确定当前目标待处理窗口对应的圆形感兴趣区域。
具体的，本步骤中将分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域之前，还可以包括判断是否存在与当前圆形感兴趣区域迭代次数相同的包含有特征的圆形感兴趣区域，即未确定为当前目标待处理窗口对应的圆形感兴趣区域的圆形感兴趣区域；若存在，则从与当前圆形感兴趣区域迭代次数相同的包含有特征的圆形感兴趣区域中选择一个圆形感兴趣区域作为当前圆形感兴趣区域；若不存在，则将当前圆形感兴趣区域分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域。
需要说明的是，图7所示的圆形四叉树迭代分裂方式，是以每次对一个圆形感兴趣区域(即当前圆形感兴趣区域)进行圆形四叉树分裂为例进行的展示，采用其他方式进行圆形四叉树的迭代分裂，如在当前目标待处理窗口中的圆形感兴趣区域的数量不大于阈值时，同时对多个不为第一圆形感兴趣区域或第二圆形感兴趣区域的相同迭代次数的圆形感兴趣区域进行圆形四叉树分裂。本实施例对此不做任何限制。
进一步的，本步骤得到每个目标待处理窗口对应的圆形感兴趣区域后，由于采用的是圆形感兴趣区域，因此会在区域周围留存一些特征，这些特征具有分散特点；本步骤可以留存目标待处理窗口对应的圆形感兴趣区域外的 特征，用于防止过度均匀化、特征过少等现象，同时可以在三维重建过程中用来矫正虚拟现实活动中的深度和位姿信息。
步骤204：对目标圆形感兴趣区域中的点特征进行特征筛选，得到每个目标圆形感兴趣区域对应的预设数量的目标点特征；其中，目标圆形感兴趣区域为存在大于预设数量的点特征的圆形感兴趣区域。
可以理解的是，本步骤的目的可以为通过对目标圆形感兴趣区域中的点特征进行特征筛选，使每个目标待处理窗口对应的圆形感兴趣区域中的点特征的数量小于或等于预设数量，从而在保留圆形感兴趣区域中的线特征约束点的情况下，减少三维重建中冗余的点特征。
具体的，对于本步骤中对目标圆形感兴趣区域中的点特征进行特征筛选，得到每个目标圆形感兴趣区域对应的预设数量的目标点特征的具体方式，可以由设计人员自行设置，如预设数量为1时，本步骤可以从当前目标圆形感兴趣区域的周围提取预设采样数量的采样点；将当前目标圆形感兴趣区域中的每个特征点的灰度梯度与采样点的灰度梯度进行对比，得到当前目标圆形感兴趣区域中的每个特征点的二进制特征质量值；根据二进制特征质量值，从当前目标圆形感兴趣区域中的全部特征点中选择一个特征点作为当前目标圆形感兴趣区域对应的目标点特征；其中，当前目标圆形感兴趣区域可以为任一包含有多个点特征的目标圆形感兴趣区域；如图8所示，预设采样数量为20时，本步骤可以从当前目标圆形感兴趣区域以外的圆上等周长提取20个采样点，将这些采样点处的灰度梯度信息(g i)与当前目标圆形感兴趣区域内每个特征点的灰度梯度信息(g feature)进行大小比对，得出每个特征点的二进制的特征质量值(即二进制特征质量值)；如通过下式确定当前目标圆形感兴趣区域中当前特征点的二进制特征质量值；
Figure PCTCN2020127362-appb-000003
上式中，g feature为当前特征点的灰度梯度，g i为第i个采样点的灰度梯度。
对应的，对于上述根据当前目标圆形感兴趣区域中每个特征点的二进制特征质量值，从当前目标圆形感兴趣区域中的全部特征点中选择一个特征点作为当前目标圆形感兴趣区域对应的目标点特征的具体方式，可以由设计人员自行设置，如可以选择二进制特征质量值中1的数量最多的特征点作为当 前目标圆形感兴趣区域对应的目标点特征；也可以选择二进制特征质量值中0的数量最多的特征点作为当前目标圆形感兴趣区域对应的目标点特征；按照其它方式选择当前目标圆形感兴趣区域对应的目标点特征。只要处理器可以利用当前目标圆形感兴趣区域中每个特征点的二进制特征质量值，确定当前目标圆形感兴趣区域对应的目标点特征，本实施例对此不做任何限制。
具体的，本步骤中的目标圆形感兴趣区域也可以为存在大于预设数量的点特征且不存在线特征约束点的圆形感兴趣区域。
步骤205：利用目标圆形感兴趣区域中的目标特征的特征匹配结果和非目标圆形感兴趣区域中的特征的特征匹配结果，完善当前图像帧对应的三维重建；其中，目标特征包括目标点特征或目标点特征和线特征约束点；非目标圆形感兴趣区域为存在小于或等于预设数量的点特征的圆形感兴趣区域。
可以理解的是，本步骤中目标圆形感兴趣区域中的目标特征可以为根据目标圆形感兴趣区域中的目标点特征和线特征约束点确定的特征，即目标圆形感兴趣区域可以包括多个目标点特征，一个目标圆形感兴趣区域中的每个目标点特征或线特征约束点可以作为该目标圆形感兴趣区域的一个目标特征；对应的，本步骤之前还可以包括根据每个目标圆形感兴趣区域中的目标点特征和线特征约束点，确定每个目标圆形感兴趣区域各自对应的目标特征的步骤。本步骤中的非目标圆形感兴趣区域中存在线特征约束点和/或小于或等于预设数量的点特征。本步骤的目的可以为利用目标圆形感兴趣区域中的目标特征和非目标圆形感兴趣区域中的特征与预设图像帧(如前一图像帧)进行特征匹配的特征匹配结果，利用特征匹配结果为匹配成功的目标特征和特征进行当前图像帧对应的三维重建，实现了当前图像帧对应的几何三维重建。
具体的，本步骤可以包括将目标圆形感兴趣区域中的目标特征和非目标圆形感兴趣区域中的特征与预设图像帧进行特征匹配，得到特征匹配结果的步骤。如预设图像帧为前一图像帧时，处理器可以利用每个待匹配特征的二进制特征质量值，将每个待匹配特征与前一图像帧进行特征匹配，得到每个待匹配特征的特征匹配结果；其中，待匹配特征包括目标圆形感兴趣区域中的目标特征和非目标圆形感兴趣区域中的特征。例如当前的待匹配特征(即当前待匹配特征)为点特征或目标点特征时，直接利用上述二进制特征质量 值计算方式得到的二进制特征质量值与前一图像帧进行特征匹配，得到当前待匹配特征的特征匹配结果。当前待匹配特征为线特征约束点时，可以利用当前待匹配特征对应的目标描述子(D p)，将当前待匹配特征与前一图像帧进行特征匹配；其中，目标描述子为第一描述子(D LBD)与第二描述子(D g)之和，即D p＝D LBD+D g，第一描述子为当前待匹配特征所对应的线特征通过LBD算法计算得到的二进制描述子，第二描述子为当前待匹配特征的二进制特征质量值，即利用上述二进制特征质量值计算方式得到的当前待匹配特征的二进制特征质量值。也就是说，步骤203中可以将每个圆形感兴趣区域中的每个特征的灰度梯度与相应的采样点的灰度梯度进行对比，得到每个圆形感兴趣区域中的每个特征的二进制特征质量值。
对应的，本步骤可以将特征匹配结果为匹配成功的待匹配特征加入三维重建特征集；将三维重建特征集中的待匹配特征划分为三维棱角参考特征和三维面参考特征，并利用三维棱角参考特征和三维面参考特征进行当前图像帧对应的三维重建；其中，三维棱角参考特征为满足对应的线特征的线性约束关系的待匹配特征，三维面参考特征为其他的特征匹配结果为匹配成功的待匹配特征。如图3所示，将匹配点(即匹配点特征匹配结果为匹配成功的待匹配特征)加入三维重建特征集合后，通过判断匹配点是否满足线约束(即对应的线特征的线性约束关系)，可以区分出匹配点中的点特征和线特征约束点，从而进行相应的参考特征的归类，提高三维重建的效率。
本实施例中，本申请实施实例通过在三维重建的特征提取阶段引入线特征，通过线特征建立线特征约束点，从而能够以一种抗遮挡和高精度的方式感知三维重建场景中的线段特征，加速简单规则三维模型的重建效率；并且本申请采用圆形区域建立图像中的感兴趣区域，并通过窗口滑动的方式进行特征提取，以四叉树的方式对三维重建中的特征进行均匀化和去冗余化处理，从而提高了三维重建的效率和精度，提升了用户体验
请参考图9，图9为本申请实施例所提供的一种三维重建装置的结构框图。该装置可以包括：
滑动计算模块10，用于根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口；
特征提取模块20，用于对待处理窗口进行点特征提取和线特征提取，确定每个待处理窗口中各自对应的特征；
迭代分裂模块30，用于对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个目标待处理窗口对应的圆形感兴趣区域；
特征筛选模块40，用于对目标圆形感兴趣区域中的特征进行特征筛选，得到每个目标圆形感兴趣区域对应的目标特征；
三维重建模块50，用于利用目标特征，进行当前图像帧对应的三维重建。
可选的，每个待处理窗口中各自对应的特征可以包括点特征和/或线特征约束点。
可选的，目标待处理窗口可以为存在对应的特征的待处理窗口，每个目标待处理窗口对应的圆形感兴趣区域的数量可以大于阈值，每个目标待处理窗口对应的圆形感兴趣区域可以包括第一圆形感兴趣区域或第一圆形感兴趣区域和第二圆形感兴趣区域，第一圆形感兴趣区域可以为只存在一个特征的圆形感兴趣区域，第二圆形感兴趣区域可以为存在线特征约束点的圆形感兴趣区域。
可选的，特征筛选模块40，可以包括：
点特征筛选子模块，用于对目标圆形感兴趣区域中的点特征进行特征筛选，得到每个目标圆形感兴趣区域中预设数量的目标点特征；其中，目标圆形感兴趣区域为存在大于预设数量的点特征的圆形感兴趣区域；
确定子模块，用于根据每个目标圆形感兴趣区域中的目标点特征和线特征约束点，确定每个目标圆形感兴趣区域各自对应的目标特征；其中，每个目标圆形感兴趣区域各自对应的目标特征包括目标点特征，或目标点特征与线特征约束点。
可选的，特征提取模块20，可以包括：
第一提取子模块，用于对当前待处理窗口进行点特征提取和线特征提取，得到当前待处理窗口对应的点特征和线特征；
第二提取子模块，用于提取每个线特征各自的对应的目标数量的线特征约束点；其中，当前线特征对应的目标数量的线特征约束点符合当前线特征的线性约束关系，当前线特征为任一线特征。
可选的，迭代分裂模块30，可以包括：
根节点确定子模块，用于根据预设窗口尺寸，确定当前目标待处理窗口的根节点；
分裂子子模块，用于以根节点为中心点，在当前目标待处理窗口进行圆形四叉树分裂；
第一判断子模块，用于判断当前圆形感兴趣区域是否为第二圆形感兴趣区域；若为第二圆形感兴趣区域，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
第二判断子模块，用于若不为第二圆形感兴趣区域，则判断当前圆形感兴趣区域是否为第一圆形感兴趣区域；为第一圆形感兴趣区域，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；其中，第一圆形感兴趣区域为只存在一个点特征的圆形感兴趣区域。
可选的，迭代分裂模块30，还可以包括：
第三判断子模块，用于若不为第一圆形感兴趣区域，则判断当前目标待处理窗口中的圆形感兴趣区域的数量是否大于阈值；若大于阈值，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
迭代子模块，用于若不大于阈值，则对当前圆形感兴趣区域进行圆形四叉树分裂，并将分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域，向第一判断子模块发送启动信号。
可选的，预设数量为1时，特征筛选模块40，可以包括：
采样子模块，用于从当前目标圆形感兴趣区域的周围提取预设采样数量的采样点；
对比子模块，用于将当前目标圆形感兴趣区域中的每个特征点的灰度梯度与采样点的灰度梯度进行对比，得到当前目标圆形感兴趣区域中的每个特征点的二进制特征质量值；
筛选子模块，用于根据二进制特征质量值，从当前目标圆形感兴趣区域中的全部特征点中选择一个特征点作为当前目标圆形感兴趣区域对应的目标点特征。
可选的，预设采样数量为20时，对比子模块可以具体用于通过
Figure PCTCN2020127362-appb-000004
确定当前目标圆形感兴趣区域中当前特征点的二进 制特征质量值；其中，g feature为当前特征点的灰度梯度，g i为第i个采样点的灰度梯度。
可选的，三维重建模块50，可以具体用于利用目标特征的特征匹配结果和非目标圆形感兴趣区域中的特征的特征匹配结果，完善当前图像帧对应的三维重建；其中，非目标圆形感兴趣区域为每个目标待处理窗口对应的圆形感兴趣区域中目标圆形感兴趣区域以外的圆形感兴趣区域。
可选的，三维重建模块50，可以包括：
特征匹配子模块，用于利用每个待匹配特征的二进制特征质量值，将每个待匹配特征与前一图像帧进行特征匹配，得到每个待匹配特征的特征匹配结果；其中，待匹配特征包括目标圆形感兴趣区域中的目标特征和非目标圆形感兴趣区域中的特征；
分配子模块，用于将特征匹配结果为匹配成功的待匹配特征加入三维重建特征集；
划分重建子模块，用于将三维重建特征集中的待匹配特征划分为三维棱角参考特征和三维面参考特征，并利用三维棱角参考特征和三维面参考特征进行当前图像帧对应的三维重建；其中，三维棱角参考特征为满足对应的线特征的线性约束关系的待匹配特征。
可选的，当前待匹配特征为线特征约束点时，特征匹配子模块可以具体用于利用当前待匹配特征对应的目标描述子，将当前待匹配特征与前一图像帧进行特征匹配；其中，目标描述子为第一描述子与第二描述子之和，第一描述子为当前待匹配特征所对应的线特征通过LBD算法计算得到的二进制描述子，第二描述子为当前待匹配特征的二进制特征质量值。
本实施例中，本申请实施实例通过在三维重建的特征提取阶段引入线特征，从而能够更好的感知三维重建场景中的线段特征，加速简单规则三维模型的重建效率；并且本申请采用圆形区域建立图像中的感兴趣区域，并通过窗口滑动的方式进行特征提取，以四叉树的方式对三维重建中的特征进行均匀化和去冗余化处理，从而提高了三维重建的效率和精度，提升了用户体验。
请参考图10，图10为本申请实施例所提供的一种三维重建设备的结构示 意图。该设备1可以包括：
存储器11，用于存储计算机程序；处理器12，用于执行该计算机程序时实现如上述实施例所提供的三维重建方法的步骤。
设备1可以具体为VR设备、AR设备、MR设备或其他需要进行三维重建的设备，设备1可以包括存储器11、处理器12和总线13。
其中，存储器11至少包括一种类型的可读存储介质，该可读存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等)、磁性存储器、磁盘、光盘等。存储器11在一些实施例中可以是设备1的内部存储单元。存储器11在另一些实施例中也可以是设备1的外部存储设备，例如AR设备、VR设备或MR设备上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。进一步地，存储器11还可以既包括设备1的内部存储单元也包括外部存储设备。存储器11不仅可以用于存储安装于设备1的应用软件及各类数据，例如：执行三维重建方法的程序的代码等，还可以用于暂时地存储已经输出或者将要输出的数据。
处理器12在一些实施例中可以是一中央处理器(Central Processing Unit,CPU)、控制器、微控制器、微处理器或其他数据处理芯片，用于运行存储器11中存储的程序代码或处理数据，例如执行三维重建方法的程序的代码等。
该总线13可以是外设部件互连标准(peripheral component interconnect，简称PCI)总线或扩展工业标准结构(extended industry standard architecture，简称EISA)总线等。该总线可以分为地址总线、数据总线、控制总线等。为便于表示，图10中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
进一步地，设备还可以包括网络接口14，网络接口14可选的可以包括有线接口和/或无线接口(如WI-FI接口、蓝牙接口等)，通常用于在该设备1与其他电子设备之间建立通信连接。
可选地，该设备1还可以包括用户接口15，用户接口15可以包括显示器(Display)、输入单元比如按键，可选的用户接口15还可以包括标准的有线接口、无线接口。可选地，在一些实施例中，显示器可以是LED显示器、液晶显示器、触控式液晶显示器以及OLED(Organic Light-Emitting Diode，有机发光二极管)触摸器等。其中，显示器也可以适当的称为显示屏或显示单 元，用于显示在设备1中处理的信息以及用于显示可视化的用户界面。
图10仅示出了具有组件11-15的设备1，本领域技术人员可以理解的是，图10示出的结构并不构成对设备1的限定，可以包括比图示更少或者更多的部件，或者组合某些部件，或者不同的部件布置。
此外，本申请实施例还公开了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，该计算机程序被处理器执行时实现如上述实施例所提供的三维重建方法的步骤。
其中，该存储介质可以包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
本说明书中各个实施例采用并列或者递进的方式描述，每个实施例重点说明的都是与其它实施例的不同之处，各个实施例之间相同或相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处可参见方法部分说明。
本领域普通技术人员还可以理解，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
结合本文中所公开的实施例描述的方法或算法的步骤可以直接用硬件、处理器执行的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。
还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗 示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。

Claims (15)

  1. 一种三维重建方法，其特征在于，包括：
    根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口；
    对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征；
    对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域；
    对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征；
    利用所述目标特征，进行当前图像帧对应的三维重建。
  2. 根据权利要求1所述的三维重建方法，其特征在于，每个所述待处理窗口中各自对应的特征包括点特征和/或线特征约束点。
  3. 根据权利要求2所述的三维重建方法，其特征在于，所述目标待处理窗口为存在对应的特征的待处理窗口，每个所述目标待处理窗口对应的圆形感兴趣区域的数量大于阈值，每个所述目标待处理窗口对应的圆形感兴趣区域包括第一圆形感兴趣区域或所述第一圆形感兴趣区域和第二圆形感兴趣区域，所述第一圆形感兴趣区域为只存在一个所述特征的圆形感兴趣区域，所述第二圆形感兴趣区域为存在所述线特征约束点的圆形感兴趣区域。
  4. 根据权利要求2所述的三维重建方法，其特征在于，所述对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征，包括：
    对所述目标圆形感兴趣区域中的所述点特征进行特征筛选，得到每个所述目标圆形感兴趣区域中预设数量的目标点特征；其中，所述目标圆形感兴趣区域为存在大于所述预设数量的所述点特征的圆形感兴趣区域；
    根据每个所述目标圆形感兴趣区域中的所述目标点特征和所述线特征约束点，确定每个所述目标圆形感兴趣区域各自对应的目标特征；其中，每个所述目标圆形感兴趣区域各自对应的目标特征包括所述目标点特征，或所述目标点特征与所述线特征约束点。
  5. 根据权利要求2所述的三维重建方法，其特征在于，所述对所述待处 理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征，包括：
    对当前待处理窗口进行点特征提取和线特征提取，得到当前待处理窗口对应的点特征和线特征；
    提取每个所述线特征各自的对应的目标数量的线特征约束点；其中，当前线特征对应的所述目标数量的线特征约束点符合当前线特征的线性约束关系，当前线特征为任一所述线特征。
  6. 根据权利要求3所述的三维重建方法，其特征在于，所述对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域，包括：
    根据所述预设窗口尺寸，确定当前目标待处理窗口的根节点；
    以所述根节点为中心点，在当前目标待处理窗口进行圆形四叉树分裂；
    判断当前圆形感兴趣区域是否为所述第二圆形感兴趣区域；
    若是，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
    若否，则判断当前圆形感兴趣区域是否为所述第一圆形感兴趣区域。
  7. 根据权利要求6所述的三维重建方法，其特征在于，若当前圆形感兴趣区域为所述第一圆形感兴趣区域，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
    若当前圆形感兴趣区域不为所述第一圆形感兴趣区域，则判断当前目标待处理窗口中的圆形感兴趣区域的数量是否大于所述阈值；
    若大于所述阈值，则将当前圆形感兴趣区域作为当前目标待处理窗口对应的圆形感兴趣区域；
    若不大于所述阈值，则对当前圆形感兴趣区域进行圆形四叉树分裂，并将分裂得到的四个圆形感兴趣区域中的一个圆形感兴趣区域作为当前圆形感兴趣区域，执行所述判断当前圆形感兴趣区域是否为所述第二圆形感兴趣区域的步骤。
  8. 根据权利要求4所述的三维重建方法，其特征在于，所述预设数量为1时，所述对目标圆形感兴趣区域中的点特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的预设数量的目标点特征，包括：
    从当前目标圆形感兴趣区域的周围提取预设采样数量的采样点；
    将当前目标圆形感兴趣区域中的每个特征点的灰度梯度与所述采样点的灰度梯度进行对比，得到当前目标圆形感兴趣区域中的每个特征点的二进制特征质量值；
    根据所述二进制特征质量值，从当前目标圆形感兴趣区域中的全部特征点中选择一个特征点作为当前目标圆形感兴趣区域对应的目标点特征。
  9. 根据权利要求8所述的三维重建方法，其特征在于，所述预设采样数量为20时，所述将当前目标圆形感兴趣区域中的每个特征点的灰度梯度与所述采样点的灰度梯度进行对比，得到当前目标圆形感兴趣区域中的每个特征点的二进制特征质量值，包括：
    通过
    Figure PCTCN2020127362-appb-100001
    确定当前目标圆形感兴趣区域中当前特征点的二进制特征质量值；其中，g feature为当前特征点的灰度梯度，g i为第i个采样点的灰度梯度。
  10. 根据权利要求1至9任一项所述的三维重建方法，其特征在于，所述利用所述目标特征，进行当前图像帧对应的三维重建，包括：
    利用所述目标特征的特征匹配结果和非目标圆形感兴趣区域中的特征的特征匹配结果，完善当前图像帧对应的三维重建；其中，所述非目标圆形感兴趣区域为每个所述目标待处理窗口对应的圆形感兴趣区域中所述目标圆形感兴趣区域以外的圆形感兴趣区域。
  11. 根据权利要求10所述的三维重建方法，其特征在于，所述利用所述目标特征，进行当前图像帧对应的三维重建，包括：
    利用每个待匹配特征的二进制特征质量值，将每个所述待匹配特征与前一图像帧进行特征匹配，得到每个所述待匹配特征的特征匹配结果；其中，所述待匹配特征包括所述目标特征；
    将所述特征匹配结果为匹配成功的所述待匹配特征加入三维重建特征集；
    将所述三维重建特征集中的所述待匹配特征划分为三维棱角参考特征和三维面参考特征，并利用所述三维棱角参考特征和所述三维面参考特征进行 当前图像帧对应的三维重建；其中，所述三维棱角参考特征为满足对应的线特征的线性约束关系的所述待匹配特征。
  12. 根据权利要求11所述的三维重建方法，其特征在于，当前待匹配特征为线特征约束点时，所述利用每个待匹配特征的二进制特征质量值，将每个所述待匹配特征与前一图像帧进行特征匹配，得到每个所述待匹配特征的特征匹配结果，包括：
    利用当前待匹配特征对应的目标描述子，将当前待匹配特征与前一图像帧进行特征匹配；其中，所述目标描述子为第一描述子与第二描述子之和，所述第一描述子为当前待匹配特征所对应的线特征通过LBD算法计算得到的二进制描述子，所述第二描述子为当前待匹配特征的二进制特征质量值。
  13. 一种三维重建装置，其特征在于，包括：
    滑动计算模块，用于根据预设窗口尺寸，对获取的当前图像帧进行滑动计算，得到待处理窗口；
    特征提取模块，用于对所述待处理窗口进行点特征提取和线特征提取，确定每个所述待处理窗口中各自对应的特征；
    迭代分裂模块，用于对每个目标待处理窗口进行圆形四叉树迭代分裂，得到每个所述目标待处理窗口对应的圆形感兴趣区域；
    特征筛选模块，用于对目标圆形感兴趣区域中的所述特征进行特征筛选，得到每个所述目标圆形感兴趣区域对应的目标特征；
    三维重建模块，用于利用所述目标特征，进行当前图像帧对应的三维重建。
  14. 一种三维重建设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述计算机程序时实现如权利要求1至12任一项所述的三维重建方法的步骤。
  15. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至12任一项所述的三维重建方法的步骤。
PCT/CN2020/127362 2020-07-17 2020-11-07 一种三维重建方法、装置、设备及计算机可读存储介质 WO2022011898A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/004,847 US20230306619A1 (en) 2020-07-17 2020-11-07 Three-dimensional reconstruction method, apparatus, and device, and computer readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010692404.7 2020-07-17
CN202010692404.7A CN111862343B (zh) 2020-07-17 2020-07-17 一种三维重建方法、装置、设备及计算机可读存储介质

Publications (1)

Publication Number Publication Date
WO2022011898A1 true WO2022011898A1 (zh) 2022-01-20

Family

ID=73001169

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/127362 WO2022011898A1 (zh) 2020-07-17 2020-11-07 一种三维重建方法、装置、设备及计算机可读存储介质

Country Status (3)

Country Link
US (1) US20230306619A1 (zh)
CN (1) CN111862343B (zh)
WO (1) WO2022011898A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114170366A (zh) * 2022-02-08 2022-03-11 荣耀终端有限公司 基于点线特征融合的三维重建方法及电子设备

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111862343B (zh) * 2020-07-17 2024-02-02 歌尔科技有限公司 一种三维重建方法、装置、设备及计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100097886A (ko) * 2009-02-27 2010-09-06 숭실대학교산학협력단 다수의 가상 평면 정보를 이용한 3차원 물체의 기하 정보 추출 기법
CN109493417A (zh) * 2018-10-31 2019-03-19 深圳大学 三维物体重建方法、装置、设备和存储介质
CN111126436A (zh) * 2019-11-21 2020-05-08 珠海格力智能装备有限公司 视觉匹配方法及装置
CN111862343A (zh) * 2020-07-17 2020-10-30 歌尔科技有限公司 一种三维重建方法、装置、设备及计算机可读存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101175439B (zh) * 2005-05-12 2010-05-26 皇家飞利浦电子股份有限公司 执行超短扫描和对最新数据的更强加权的连续计算机层析成像
CN109308737A (zh) * 2018-07-11 2019-02-05 重庆邮电大学 一种三阶段式点云配准方法的移动机器人v-slam方法
CN110044354B (zh) * 2019-03-28 2022-05-20 东南大学 一种双目视觉室内定位与建图方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100097886A (ko) * 2009-02-27 2010-09-06 숭실대학교산학협력단 다수의 가상 평면 정보를 이용한 3차원 물체의 기하 정보 추출 기법
CN109493417A (zh) * 2018-10-31 2019-03-19 深圳大学 三维物体重建方法、装置、设备和存储介质
CN111126436A (zh) * 2019-11-21 2020-05-08 珠海格力智能装备有限公司 视觉匹配方法及装置
CN111862343A (zh) * 2020-07-17 2020-10-30 歌尔科技有限公司 一种三维重建方法、装置、设备及计算机可读存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114170366A (zh) * 2022-02-08 2022-03-11 荣耀终端有限公司 基于点线特征融合的三维重建方法及电子设备

Also Published As

Publication number Publication date
CN111862343A (zh) 2020-10-30
US20230306619A1 (en) 2023-09-28
CN111862343B (zh) 2024-02-02

Similar Documents

Publication Publication Date Title
US11842438B2 (en) Method and terminal device for determining occluded area of virtual object
US9251613B2 (en) Systems and methods for automatically applying effects based on media content characteristics
WO2018103608A1 (zh) 一种文字检测方法、装置及存储介质
WO2022011898A1 (zh) 一种三维重建方法、装置、设备及计算机可读存储介质
JP2018519574A (ja) テキスト画像処理方法および装置
CN110503682B (zh) 矩形控件识别方法、装置、终端及存储介质
US10152213B2 (en) Techniques for selecting objects in images
JP7264929B2 (ja) 背景なし画像の生成方法及び装置、電子機器、記憶媒体並びにコンピュータプログラム
WO2020125062A1 (zh) 一种图像融合方法及相关装置
CN108960012B (zh) 特征点检测方法、装置及电子设备
CN113408251B (zh) 版式文档的处理方法、装置、电子设备及可读存储介质
WO2022227218A1 (zh) 药名识别方法、装置、计算机设备和存储介质
US8494284B2 (en) Methods and apparatuses for facilitating detection of text within an image
CN111340954A (zh) 户型墙体绘制方法及其模型训练的方法、装置
CN111339944A (zh) 装修风格识别方法、装置及电子设备
US8824778B2 (en) Systems and methods for depth map generation
AU2011265380B2 (en) Determining transparent fills based on a reference background colour
US11055526B2 (en) Method, system and apparatus for processing a page of a document
US9613288B2 (en) Automatically identifying and healing spots in images
CN116259064B (zh) 表格结构识别方法、表格结构识别模型的训练方法及装置
US11281935B2 (en) 3D object detection from calibrated 2D images
CN114387315A (zh) 图像处理模型训练、图像处理方法、装置、设备及介质
CN111915703B (zh) 一种图像生成方法和装置
CN113657317A (zh) 一种货物位置识别方法、系统、电子设备及存储介质
CN104112135B (zh) 文本图像提取装置以及方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20945036

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20945036

Country of ref document: EP

Kind code of ref document: A1