WO2022237026A1 - 平面信息检测方法及系统 - Google Patents

平面信息检测方法及系统 Download PDF

Info

Publication number
WO2022237026A1
WO2022237026A1 PCT/CN2021/118287 CN2021118287W WO2022237026A1 WO 2022237026 A1 WO2022237026 A1 WO 2022237026A1 CN 2021118287 W CN2021118287 W CN 2021118287W WO 2022237026 A1 WO2022237026 A1 WO 2022237026A1
Authority
WO
WIPO (PCT)
Prior art keywords
plane
information
point cloud
plane information
planar
Prior art date
Application number
PCT/CN2021/118287
Other languages
English (en)
French (fr)
Inventor
吴涛
Original Assignee
青岛小鸟看看科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 青岛小鸟看看科技有限公司 filed Critical 青岛小鸟看看科技有限公司
Priority to US17/816,377 priority Critical patent/US11741621B2/en
Publication of WO2022237026A1 publication Critical patent/WO2022237026A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker

Definitions

  • the present disclosure relates to the technical field of plane detection, and more specifically, to a plane information detection method and system.
  • the main application scenario is that when users interact with VR/AR/MR scenes, they can be automatically identified by the multi-eye tracking camera on the head. Track some behavior trajectory information of the user's hand, and detect gesture instructions through some behavior trajectory of the hand, and then use it as the input information of the artificial virtual reality system to interact with the virtual scene.
  • the point cloud data of the physical environment is mainly generated by the computing processor, and the plane detection and plane fitting are performed on the point cloud data, and then various plane information in the physical environment are obtained.
  • Existing methods have insufficient or excessive plane detection for some scenes in the physical environment, which affects the accuracy and stability of plane detection.
  • the purpose of the present disclosure is to provide a plane information detection method and system to solve the current problems of plane detection, such as insufficient or excessive detection, which affects detection accuracy and stability.
  • the planar information detection method provided in this disclosure includes obtaining point cloud information in the user’s physical environment; performing iterative regression on point cloud information to fit all planar information corresponding to the physical environment; merging all planar information with preset rules , to obtain the combined plane information set; perform plane segmentation on the plane information set based on the pre-trained plane segmentation model, and obtain the segmented plane information; filter the segmented plane information to determine all target planes corresponding to the physical environment information.
  • the training process of the plane segmentation model includes: collecting plane image data, and performing data labeling on the plane image data to obtain a plane image label set; training the neural network model based on the plane image label set, until the neural network The network model converges within a preset range, forming a planar segmentation model.
  • the plane information includes all point cloud information in the current plane and the two-dimensional image information corresponding to each point cloud information; the point cloud information is obtained through the head-mounted positioning tracking module in the virtual reality display device .
  • the iterative regression process of the point cloud information includes: performing RANSAC algorithm processing on the point cloud information, and fitting all plane information corresponding to the physical environment; wherein, the fitted minimum point cloud information The number is 10.
  • the process of fitting all plane information corresponding to the physical environment includes: randomly obtaining a preset number of nearest point clouds from the point cloud information, and performing plane fitting of the RANSAC algorithm , to obtain the initial plane; obtain the normal vector of the initial plane; randomly obtain the random point cloud in the point cloud information, and perform plane fitting on the random point cloud and the point cloud of the initial plane to obtain a new normal vector; judge the initial plane The difference between the normal vector and the new normal vector, if the difference meets the preset range, repeat the above iterative steps, otherwise, keep the plane information of the last iteration until all the plane information is obtained.
  • the preset range is 1cm-4cm.
  • the process of merging all plane information with preset rules includes: judging the inclination angle between any two plane information in all plane information, and when the inclination angle satisfies the first threshold value, judging the inclination angle between the two plane information Whether the number of point clouds whose distance between point clouds in three-dimensional space on the information is within a preset value satisfies the second threshold; when both the first threshold and the second threshold are satisfied, the information of the two planes is merged.
  • the first threshold is 4°-12°; the second threshold is 8-20.
  • the process of screening the segmented plane information includes: obtaining the ratio of the number of pixels in each plane in the segmented plane information to the corresponding plane in the plane information set; when the ratio When the preset threshold is met, it is determined that the plane information is the target plane information in the physical environment.
  • a plane information detection system including: a point cloud information acquisition unit configured to acquire point cloud information in the user's physical environment; a plane information fitting unit configured to iterate the point cloud information Regression, fitting all the plane information corresponding to the physical environment; the plane information set acquisition unit is set to merge all plane information with preset rules to obtain the merged plane information set; the segmentation processing unit is set to The trained plane segmentation model performs plane segmentation on the plane information set to obtain the divided plane information; the target plane information determination unit is set to filter the segmented plane information and determine all target plane information corresponding to the physical environment.
  • the plane information detection method and system to obtain point cloud information in the user's physical environment; perform iterative regression on the point cloud information to fit all plane information corresponding to the physical environment; merge all plane information with preset rules, to obtain the merged plane information set; based on the pre-trained plane segmentation model, the plane information set is plane-segmented to obtain the divided plane information; the divided plane information is screened to determine all target plane information corresponding to the physical environment , it can stably extract the plane information in the indoor environment, the extraction accuracy is high, and the applicable range is wide.
  • FIG. 1 is a flow chart of a plane information detection method according to an embodiment of the disclosure
  • FIG. 2 is a logical block diagram of a planar information detection system according to an embodiment of the present disclosure.
  • Fig. 1 shows the flow of a plane information detection method according to an embodiment of the present disclosure.
  • the planar information detection method of the embodiment of the present disclosure mainly includes the following steps:
  • S140 Carry out plane segmentation on the plane information set based on the pre-trained plane segmentation model, and obtain the divided plane information;
  • S150 Screen the segmented plane information to determine all target plane information corresponding to the physical environment.
  • the planar information detection method of the embodiment of the present disclosure mainly includes two stages, one stage is the training stage of the planar segmentation model, and the other stage is the stage of obtaining the target planar information based on the planar segmentation model.
  • the two stages are described separately.
  • the training process of the plane segmentation model includes: collecting plane image data and labeling the plane image data to obtain a plane image label set; training the neural network model based on the plane image label set until the neural network model converges to the preset In the range, a plane segmentation model is formed.
  • a virtual reality display device can be used to collect planar image data in various scenarios, such as planar image data such as office desktops, floors, and conference desktops in an office scene, planar image data in a living room environment at home, For example, planar image data in other environmental scenes such as living room table, living room floor, living room coffee table and other planar image data, a total of 3 million planar image data are collected to form a planar image data set, and then planar data is processed for each image data in the data set Annotate to obtain the corresponding plane image label set; then, train the plane segmentation model based on the neural network model until a high-precision plane segmentation model is obtained.
  • planar image data such as office desktops, floors, and conference desktops in an office scene
  • planar image data in a living room environment at home planar image data in other environmental scenes such as living room table, living room floor, living room coffee table and other planar image data
  • planar data is processed for each image data in the data set Annotate to obtain the corresponding plane
  • a head-mounted positioning tracking sensor is set in the above-mentioned virtual reality display device (HMD), and the head-mounted positioning tracking sensor is usually a fisheye barefoot camera, and two or more number of fisheye wide-angle cameras.
  • the training of the plane segmentation model can be performed through the acquired image data of the fisheye wide-angle camera built in the HMD.
  • the point cloud information in the user's physical environment can be obtained through the head-mounted positioning tracking sensor of the HMD.
  • the point cloud mainly refers to the mass point collection of the surface characteristics of the target.
  • the point cloud obtained according to the principle of laser measurement includes three-dimensional coordinates and Laser reflection intensity
  • the point cloud obtained according to the principle of photogrammetry includes three-dimensional coordinates and color information.
  • the point cloud information in this disclosure may include massive three-dimensional coordinates, color information and laser reflection intensity of the surface characteristics of the target.
  • iterative regression is performed on the obtained point cloud information to fit all the plane information corresponding to the physical environment; where the plane information includes all point cloud information in the current plane and the two-dimensional images corresponding to each point cloud information Information and point cloud information are obtained through the head-mounted positioning tracking module in the virtual reality display device.
  • the process of performing iterative regression on point cloud information may include: performing RANSAC algorithm processing on point cloud information, and fitting all plane information corresponding to the physical environment; wherein, in order to improve the accuracy and stability of plane fitting, fitting The number of combined minimum point cloud information can be set to 10.
  • the process of fitting all plane information corresponding to the physical environment includes: randomly obtaining a preset number of nearest point clouds in the point cloud information, and performing plane fitting of the RANSAC algorithm (for example, randomly obtaining 10 point clouds with the closest distance to each other for plane fitting) to obtain the initial plane; then, obtain the normal vector of the initial plane; randomly obtain the random point cloud in the point cloud information, and perform random point cloud and point cloud of the initial plane.
  • RANSAC algorithm for example, randomly obtaining 10 point clouds with the closest distance to each other for plane fitting
  • Plane fitting to obtain a new normal vector; judge the difference between the normal vector of the initial plane and the new normal vector, if the difference meets the preset range, such as 1cm to 4cm, repeat the above iterative steps, and continue in the point cloud information ( Randomly obtain a point cloud from the pool), and continue to do plane fitting; otherwise, if the difference is greater than the preset range, keep the plane information of the last iteration; then, continue to randomly obtain the closest distance from the preset number in the point cloud information Plane fitting is performed on the point cloud until all plane information is obtained to form a plane information set.
  • the preset range such as 1cm to 4cm
  • the process of merging all plane information with preset rules includes: judging the inclination angle between any two plane information in all plane information, and judging the inclination angle between any two plane information when the inclination angle satisfies the first threshold Whether the number of point clouds whose distance between point clouds in three-dimensional space is within a preset value satisfies the second threshold; furthermore, when both the first threshold and the second threshold are satisfied, the information of the two planes is merged.
  • the first threshold can be set to 4°-12°; the second threshold can be set to 8-20.
  • the inclination angle between any two plane information is within 8° and the distance between the nearest three-dimensional space point clouds on the two plane information is within 10cm. If the number of point clouds exceeds 10 , after merging the two plane information and obtaining a merged plane information, in the process of merging, the point cloud information corresponding to the two plane information can be combined together, and then a plane fitting is performed to obtain The merged plane information.
  • step S140 all plane information in the plane information set is plane-segmented through the plane segmentation model trained. Because there may be a false detection probability that it is not a plane before the segmentation, that is, the physical environment obtained in the above steps There may be a possibility of false detection of the plane, so the above-mentioned detected plane information can be further confirmed by combining semantic information through plane segmentation, so as to accurately obtain plane information in the real physical environment.
  • the segmented plane information is further screened, and the screening process includes: obtaining the proportion of the number of pixels of each plane in the segmented plane information in the plane corresponding to the plane information set; When the proportion satisfies the preset threshold, it is determined that the plane information is the target plane information in the physical environment.
  • calculate the proportion of the number of pixels of each plane in the divided plane information in the plane corresponding to the plane information set if the proportion is greater than 85% (the proportion can be set according to specific application scenarios or requirements and Adjustment), confirm that the corresponding plane is the plane information in the physical environment, which can be used as the target plane information.
  • all the divided plane information is screened to obtain all the target planes in the corresponding physical environment information.
  • the pixel point corresponds to a two-dimensional point
  • the spatial point cloud corresponds to a three-dimensional point in space
  • each pixel corresponds to a three-dimensional point cloud in space.
  • FIG. 2 shows a schematic logic of a plane information detection system according to an embodiment of the present disclosure.
  • planar information detection system 200 of the embodiment of the present disclosure may include:
  • the point cloud information acquisition unit 210 is configured to acquire point cloud information in the user's physical environment
  • the plane information fitting unit 220 is configured to perform iterative regression on the point cloud information to fit all the plane information corresponding to the physical environment;
  • the planar information set acquisition unit 230 is configured to merge all planar information with preset rules to obtain a merged planar information set;
  • the segmentation processing unit 240 is configured to perform plane segmentation on the plane information set based on the pre-trained plane segmentation model, and obtain the divided plane information;
  • the target plane information determining unit 250 is configured to filter the divided plane information to determine all target plane information corresponding to the physical environment.
  • the plane detection and extraction in the physical environment can be realized, and the plane detection accuracy and stability in the physical environment can be improved.
  • each module or each step of the above-mentioned disclosure can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network composed of multiple computing devices In fact, they can be implemented in program code executable by a computing device, and thus, they can be stored in a storage device to be executed by a computing device, and in some cases, can be executed in an order different from that shown here. Or described steps, or they are fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present disclosure is not limited to any specific combination of hardware and software.
  • planar information detection method has the following beneficial effects: using the above-mentioned planar information detection method and system to obtain point cloud information in the user's physical environment; iterative regression is performed on the point cloud information to fit All plane information corresponding to the physical environment; merge all plane information with preset rules to obtain the merged plane information set; perform plane segmentation on the plane information set based on the pre-trained plane segmentation model to obtain the divided plane information; Screen the segmented plane information to determine all the target plane information corresponding to the physical environment, and can stably extract the plane information in the indoor environment, with high extraction accuracy and a wide range of applications.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

一种平面信息检测方法及系统，其中的方法包括：获取用户物理环境下的点云信息（S110）；对点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息（S120）；对所有平面信息进行预设规则的合并，以获取合并后的平面信息集（S130）；基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信息（S140）；对分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息（S150）。利用上述方法能够提高物理环境下的平面检测和提取的精度和稳定性。

Description

平面信息检测方法及系统 技术领域
本公开涉及平面检测技术领域，更为具体地，涉及一种平面信息检测方法及系统。
背景技术
目前，VR/AR/MR一体机设备越来越多的进入到人们生活中，其主要的应用场景是当用户进行VR/AR/MR场景交互时，通过头戴上的多目追踪摄像头自动识别跟踪用户手一些行为轨迹信息，并通过手的一些行为轨迹检测手势指令，然后作为人造虚拟现实系统的输入信息，和虚拟场景进行交互。
现有的VR、AR或者MR均是人基于虚拟场景物体和物理环境物体进行交互，其中最为基础和最为常见的交互物体是平面，平面检测的精度往往代表了VR、AR或者MR人机交互的基础体验，由于不同的物理环境，例如办公室场所、商场场所或者家里客厅、卧室等不同环境下的场所，使其平面环境变的多种多样，其平面检测的难度和复杂度也的越来越高。
目前在VR、AR或者MR领域中，主要是通过计算处理器生成物理环境的点云数据，通过对点云数据进行平面检测和平面拟合，然后获取物理环境中的各种平面信息，但是，现有方法存在对物理环境下的一些场景的平面检测不充分，或者过充分的情况，影响平面检测精度和稳定性。
发明内容
鉴于上述问题，本公开的目的是提供一种平面信息检测方法及系统，解决目前平面检测存在的检测不充分或过充分，影响检测精度和稳定性等问题。
本公开提供的平面信息检测方法，包括获取用户物理环境下的点云信息；对点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息；对所有平面信息进行预设规则的合并，以获取合并后的平面信息集；基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信 息；对分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息。
此外，可选的技术方案是，平面分割模型的训练过程包括：采集平面图像数据，并对平面图像数据进行数据标注，以获取平面图像标注集；基于平面图像标注集训练神经网络模型，直至神经网络模型收敛在预设范围内，形成平面分割模型。
此外，可选的技术方案是，平面信息包括当前平面中所有的点云信息以及与各点云信息分别对应的二维图像信息；点云信息通过虚拟现实显示装置内的头戴定位追踪模块获取。
此外，可选的技术方案是，对点云信息进行迭代回归的过程包括：对点云信息进行RANSAC算法处理，拟合与物理环境对应的所有平面信息；其中，拟合的最小点云信息的个数为10。
此外，可选的技术方案是，拟合出与物理环境对应的所有平面信息的过程包括：在点云信息中随机获取预设个数的距离最近的点云，并进行RANSAC算法的平面拟合，以获取初始平面；获取初始平面的法向量；随机获取点云信息中的随机点云，并对随机点云和初始平面的点云进行平面拟合，获取新的法向量；判断初始平面的法向量和新的法向量之间的差异，如果差异符合预设范围，则重复上述迭代步骤，否则，保留最后一次迭代的平面信息，直至获取所有平面信息。
此外，可选的技术方案是，预设范围为1cm～4cm。
此外，可选的技术方案是，对所有平面信息进行预设规则的合并的过程包括：判断所有平面信息中任意两平面信息之间的倾斜角，当倾斜角满足第一阈值时，判断两平面信息上的三维空间点云之间的距离在预设值以内的点云的个数是否满足第二阈值；当第一阈值和第二阈值均满足时，对两平面信息进行合并。
此外，可选的技术方案是，第一阈值为4°～12°；第二阈值为8～20个。
此外，可选的技术方案是，对分割后的平面信息进行筛选的过程包括： 获取分割后的平面信息中各平面的像素点个数在平面信息集中对应的平面中的占比率；当占比率满足预设阈值时，确定平面信息为物理环境下的目标平面信息。
根据本公开的另一方面，提供一种平面信息检测系统，包括：点云信息获取单元，设置为获取用户物理环境下的点云信息；平面信息拟合单元，设置为对点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息；平面信息集获取单元，设置为对所有平面信息进行预设规则的合并，以获取合并后的平面信息集；分割处理单元，设置为基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信息；目标平面信息确定单元，设置为对分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息。
利用上述平面信息检测方法及系统，获取用户物理环境下的点云信息；对点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息；对所有平面信息进行预设规则的合并，以获取合并后的平面信息集；基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信息；对分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息，能够稳定的提取屋里环境下的平面信息，提取准确度高、可适用范围广。
为了实现上述以及相关目的，本公开的一个或多个方面包括后面将详细说明的特征。下面的说明以及附图详细说明了本公开的某些示例性方面。然而，这些方面指示的仅仅是可使用本公开的原理的各种方式中的一些方式。此外，本公开旨在包括所有这些方面以及它们的等同物。
附图说明
通过参考以下结合附图的说明，并且随着对本公开的更全面理解，本公开的其它目的及结果将更加明白及易于理解。在附图中：
图1为根据本公开实施例的平面信息检测方法的流程图；
图2为根据本公开实施例的平面信息检测系统的逻辑方框图。
在所有附图中相同的标号指示相似或相应的特征或功能。
具体实施方式
在下面的描述中，出于说明的目的，为了提供对一个或多个实施例的全面理解，阐述了许多具体细节。然而，很明显，也可以在没有这些具体细节的情况下实现这些实施例。在其它例子中，为了便于描述一个或多个实施例，公知的结构和设备以方框图的形式示出。
在本公开的描述中，需要理解的是，术语“中心”、“纵向”、“横向”、“长度”、“宽度”、“厚度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”“内”、“外”、“顺时针”、“逆时针”、“轴向”、“径向”、“周向”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本公开和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本公开的限制。
为详细描述本公开实施例的平面信息检测方法及系统，以下将结合附图对本公开的具体实施例进行详细描述。
图1示出了根据本公开实施例的平面信息检测方法的流程。
如图1所示，本公开实施例的平面信息检测方法，主要包括以下步骤：
S110：取用户物理环境下的点云信息；
S120：对所述点云信息进行迭代回归，拟合出与所述物理环境对应的所有平面信息；
S130：对所述所有平面信息进行预设规则的合并，以获取合并后的平面信息集；
S140：基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信息；
S150：对分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息。
其中，本公开实施例的平面信息检测方法主要包括两个阶段，一个阶段为平面分割模型的训练阶段，另一个阶段为基于平面分割模型获取目标平面信息的阶段，以下将结合具体实施例对上述两个阶段进行分别描述。
具体地，平面分割模型的训练过程包括：采集平面图像数据，并对平面图像数据进行数据标注，以获取平面图像标注集；基于平面图像标注集训练神经网络模型，直至神经网络模型收敛在预设范围内，形成平面分割模型。
作为具体示例，可通过虚拟现实显示设备(HMD)采集各种场景下的平面图像数据，比如办公室场景下的办公桌面、地面、会议桌面等平面图像数据，家里的客厅环境下的平面图像数据，比如客厅餐桌、客厅地面或者客厅茶几等平面图像数据等其他环境场景下的平面图像数据，共采集300万张平面图像数据，形成平面图像数据集，然后对数据集中每一张图像数据进行平面数据标注，获取对应的平面图像标注集；然后，基于神经网络模型进行平面分割模型的训练，直至获取高精度的平面分割模型。
进一步地，在上述虚拟现实显示设备(HMD)内设置有头戴定位追踪传感器，该头戴定位追踪传感器通常为鱼眼光脚摄像机，在虚拟现实显示设备(HMD)内一般设置有两个及以上个数的鱼眼广角摄像机。例如，在本公开的平面信息检测方法中，可通过获取的HMD内置的鱼眼广角摄像机的图像数据进行平面分割模型的训练。
上述步骤S110中，可通过HMD的头戴定位追踪传感器获取用户物理环境下的点云信息，点云主要是指目标表面特性的海量点集合，根据激光测量原理得到的点云，包括三维坐标和激光反射强度，而根据摄影测量原理得到的点云，包括三维坐标和颜色信息，本公开中的点云信息可包括目标表面特性的海量的三维坐标、颜色信息和激光反射强度。
进一步地，对获取的点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息；其中，平面信息包括当前平面中所有的点云信息以及与各点云信息分别对应的二维图像信息，点云信息通过虚拟现实显示装置内的头戴定位追踪模块获取。
作为具体示例，对点云信息进行迭代回归的过程可包括：对点云信息进行RANSAC算法处理，拟合与物理环境对应的所有平面信息；其中，为了提高平面拟合的精度和稳定性，拟合的最小点云信息的个数可设置为 10。
进一步地，拟合出与物理环境对应的所有平面信息的过程包括：在点云信息中随机获取预设个数的距离最近的点云，并进行RANSAC算法的平面拟合(例如，随机获取10个相互距离最近的点云进行平面拟合)，以获取初始平面；然后，获取初始平面的法向量；随机获取点云信息中的随机点云，并对随机点云和初始平面的点云进行平面拟合，获取新的法向量；判断初始平面的法向量和新的法向量之间的差异，如果差异符合预设范围，例如1cm～4cm，则重复上述迭代步骤，继续在点云信息(池)中随机获取一个点云，继续做平面拟合；否则，如何差异大于预设范围，则保留最后一次迭代的平面信息；然后，继续在点云信息中随机获取预设个数的距离最近的点云进行平面拟合，直至获取所有的平面信息，形成平面信息集。
在上述步骤S130中，对所有平面信息进行预设规则的合并的过程包括：判断所有平面信息中任意两平面信息之间的倾斜角，当倾斜角满足第一阈值时，判断两平面信息上的三维空间点云之间的距离在预设值以内的点云的个数是否满足第二阈值；进而，当第一阈值和第二阈值均满足时，对两平面信息进行合并。
在本公开的一个具体实施方式中，第一阈值可设置为4°～12°；第二阈值可设置为8～20个。
作为示例，针对所有平面信息，判断任意两平面信息之间的倾斜角在8°以内并且两个平面信息上的最近三维空间点云之间的距离在10cm以内的点云个数如果超过10个，在对该两个平面信息进行合并，并获取合并后的一个平面信息，在合并过程中，可将两个平面信息对应的点云信息组合在一起，然后进行一次平面拟合，即可获取合并后的平面信息。
然后，按照上述预设规则的判断标准，对检测出的所有平面进行判断合并，获取对应物理环境下所有平面信息，作为合并后的平面信息集。
上述步骤S140中，通过训练完成的平面分割模型对平面信息集中的所有平面信息进行平面分割，由于在进行分割前，可能存在不是平面的误检测概率，即在上述步骤中获取的物理环境中的平面可能存在误检测的可 能，为此可通过平面分割对上述检测到的平面信息进一步结合语义信息进行确认，以精准获取真实物理环境下的平面信息。
在获取分割后的平面信息后，进一步对分割后的平面信息进行筛选，筛选的过程包括：获取分割后的平面信息中各平面的像素点个数在平面信息集中对应的平面中的占比率；当占比率满足预设阈值时，确定平面信息为物理环境下的目标平面信息。
例如，计算分割后的平面信息中每个平面的像素点个数在平面信息集中对应的平面中的占比率，如果占比率大于85％(该占比率可根据具体的应用场景或需求进行设置及调整)，确认对应的平面为物理环境中的平面信息，即可作为目标平面信息，按照上述筛选方式，对分割后的所有平面信息均进行筛选，以获取对应的物理环境下的所有的目标平面信息。
其中，像素点对应二维点，空间点云对应空间三维点，每一个像素点对应一个空间三维点云。
与上述平面信息检测方法相对应，本公开还提供一种平面信息检测系统。具体地，图2示出了根据本公开实施例的平面信息检测系统的示意逻辑。
如图2所示，本公开实施例的平面信息检测系统200，可以包括：
点云信息获取单元210，设置为获取用户物理环境下的点云信息；
平面信息拟合单元220，设置为对点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息；
平面信息集获取单元230，设置为对所有平面信息进行预设规则的合并，以获取合并后的平面信息集；
分割处理单元240，设置为基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信息；
目标平面信息确定单元250，设置为对分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息。
需要说明的是，上述平面信息检测系统的实施例可参考平面信息检测方法实施例中的描述，此处不再一一赘述。
利用上述根据本公开提供的平面信息检测方法及系统，结合人工神经网络模型的语义分割技术，实现对物理环境下的平面的检测和提取，能够提高物理环境下的平面检测精度和稳定性。
如上参照附图以示例的方式描述根据本公开的平面信息检测方法及系统。但是，本领域技术人员应当理解，对于上述本公开所提出的平面信息检测方法及系统，还可以在不脱离本公开内容的基础上做出各种改进。因此，本公开的保护范围应当由所附的权利要求书的内容确定。
本实施例中的具体示例可以参考上述实施例及示例性实施方式中所描述的示例，本实施例在此不再赘述。
显然，本领域的技术人员应该明白，上述的本公开的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本公开不限制于任何特定的硬件和软件结合。
以上所述仅为本公开的优选实施例而已，并不设置为限制本公开，对于本领域的技术人员来说，本公开可以有各种更改和变化。凡在本公开的原则之内，所作的任何修改、等同替换、改进等，均应包含在本公开的保护范围之内。
工业实用性
如上所述，本公开实施例提供的平面信息检测方法具有以下有益效果：利用上述平面信息检测方法及系统，获取用户物理环境下的点云信息；对点云信息进行迭代回归，拟合出与物理环境对应的所有平面信息；对所有平面信息进行预设规则的合并，以获取合并后的平面信息集；基于预训练的平面分割模型对平面信息集进行平面分割，获取分割后的平面信息；对 分割后的平面信息进行筛选，确定与物理环境对应的所有目标平面信息，能够稳定的提取屋里环境下的平面信息，提取准确度高、可适用范围广。

Claims (12)

  1. 一种平面信息检测方法，包括：
    获取用户物理环境下的点云信息；
    对所述点云信息进行迭代回归，拟合出与所述物理环境对应的所有平面信息；
    对所述所有平面信息进行预设规则的合并，以获取合并后的平面信息集；
    基于预训练的平面分割模型对所述平面信息集进行平面分割，获取分割后的平面信息；
    对分割后的平面信息进行筛选，确定与所述物理环境对应的所有目标平面信息。
  2. 如权利要求1所述的平面信息检测方法，其中，所述平面分割模型的训练过程包括：
    采集平面图像数据，并对所述平面图像数据进行数据标注，以获取平面图像标注集；
    基于所述平面图像标注集训练神经网络模型，直至所述神经网络模型收敛在预设范围内，形成所述平面分割模型。
  3. 如权利要求1所述的平面信息检测方法，其中，
    所述平面信息包括当前平面中所有的点云信息以及与各点云信息分别对应的二维图像信息；
    所述点云信息通过虚拟现实显示装置内的头戴定位追踪模块获取。
  4. 如权利要求1所述的平面信息检测方法，其中，对所述点云信息进行迭代回归的过程包括：
    对所述点云信息进行RANSAC算法处理，拟合与所述物理环境对应的所有平面信息；其中，
    所述拟合的最小点云信息的个数为10。
  5. 如权利要求4所述的平面信息检测方法，其中，所述拟合出与所述物理环境对应的所有平面信息的过程包括：
    在所述点云信息中随机获取预设个数的距离最近的点云，并进行 RANSAC算法的平面拟合，以获取初始平面；
    获取所述初始平面的法向量；
    随机获取所述点云信息中的随机点云，并对所述随机点云和所述初始平面的点云进行平面拟合，获取新的法向量；
    判断所述初始平面的法向量和所述新的法向量之间的差异，如果所述差异符合预设范围，则重复上述迭代步骤，否则，保留最后一次迭代的平面信息，直至获取所述所有平面信息。
  6. 如权利要求5所述的平面信息检测方法，其中，
    所述预设范围为1cm～4cm。
  7. 如权利要求1所述的平面信息检测方法，其中，对所述所有平面信息进行预设规则的合并的过程包括：
    判断所述所有平面信息中任意两平面信息之间的倾斜角，当所述倾斜角满足第一阈值时，判断所述两平面信息上的三维空间点云之间的距离在预设值以内的点云的个数是否满足第二阈值；
    当所述第一阈值和所述第二阈值均满足时，对所述两平面信息进行合并。
  8. 如权利要求7所述的平面信息检测方法，其中，
    所述第一阈值为4°～12°；
    所述第二阈值为8～20个。
  9. 如权利要求1所述的平面信息检测方法，其中，对分割后的平面信息进行筛选的过程包括：
    获取所述分割后的平面信息中各平面的像素点个数在所述平面信息集中对应的平面中的占比率；
    当所述占比率满足预设阈值时，确定所述平面信息为所述物理环境下的目标平面信息。
  10. 一种平面信息检测系统，包括：
    点云信息获取单元，设置为获取用户物理环境下的点云信息；
    平面信息拟合单元，设置为对所述点云信息进行迭代回归，拟合出与 所述物理环境对应的所有平面信息；
    平面信息集获取单元，设置为对所述所有平面信息进行预设规则的合并，以获取合并后的平面信息集；
    分割处理单元，设置为基于预训练的平面分割模型对所述平面信息集进行平面分割，获取分割后的平面信息；
    目标平面信息确定单元，设置为对分割后的平面信息进行筛选，确定与所述物理环境对应的所有目标平面信息。
  11. 一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，所述计算机程序被处理器执行时实现所述权利要求1至9任一项中所述的方法，或者实现权利要求6-10任一项中所述的方法。
  12. 一种电子装置，包括存储器和处理器，所述存储器中存储有计算机程序，所述处理器被设置为运行所述计算机程序以执行所述权利要求1至9任一项中所述的方法，或者执行权利要求6-10任一项中所述的方法。
PCT/CN2021/118287 2021-05-10 2021-09-14 平面信息检测方法及系统 WO2022237026A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/816,377 US11741621B2 (en) 2021-05-10 2022-07-29 Method and system for detecting plane information

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110505159.9A CN113240678B (zh) 2021-05-10 2021-05-10 平面信息检测方法及系统
CN202110505159.9 2021-05-10

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/816,377 Continuation US11741621B2 (en) 2021-05-10 2022-07-29 Method and system for detecting plane information

Publications (1)

Publication Number Publication Date
WO2022237026A1 true WO2022237026A1 (zh) 2022-11-17

Family

ID=77133193

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/118287 WO2022237026A1 (zh) 2021-05-10 2021-09-14 平面信息检测方法及系统

Country Status (2)

Country Link
CN (1) CN113240678B (zh)
WO (1) WO2022237026A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117635888A (zh) * 2023-12-07 2024-03-01 腾讯科技（深圳）有限公司 一种数据处理方法及相关装置

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11741621B2 (en) 2021-05-10 2023-08-29 Qingdao Pico Technology Co., Ltd. Method and system for detecting plane information
CN113240678B (zh) * 2021-05-10 2023-05-30 青岛小鸟看看科技有限公司 平面信息检测方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110458805A (zh) * 2019-03-26 2019-11-15 华为技术有限公司 一种平面检测方法、计算设备以及电路系统
CN111639682A (zh) * 2020-05-13 2020-09-08 北京三快在线科技有限公司 基于点云数据的地面分割方法及装置
US20200402300A1 (en) * 2019-06-21 2020-12-24 Harbin Institute Of Technology Terrain modeling method that fuses geometric characteristics and mechanical charateristics, computer readable storage medium, and terrain modeling system thereof
CN112329946A (zh) * 2020-11-10 2021-02-05 广州极飞科技有限公司 地面分割学习模型训练方法、地平面确定方法及相关装置
CN113240678A (zh) * 2021-05-10 2021-08-10 青岛小鸟看看科技有限公司 平面信息检测方法及系统

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110070570B (zh) * 2019-03-20 2023-05-26 重庆邮电大学 一种基于深度信息的障碍物检测系统及方法
CN109993783B (zh) * 2019-03-25 2020-10-27 北京航空航天大学 一种面向复杂三维建筑物点云的屋顶及侧面优化重建方法
CN110197215A (zh) * 2019-05-22 2019-09-03 深圳市牧月科技有限公司 一种自主驾驶的地面感知点云语义分割方法
CN111507982B (zh) * 2019-06-28 2022-04-26 浙江大学 一种基于深度学习的点云语义分割方法
CN110264468B (zh) * 2019-08-14 2019-11-19 长沙智能驾驶研究院有限公司 点云数据标注、分割模型确定、目标检测方法及相关设备
CN112396650B (zh) * 2020-03-30 2023-04-07 青岛慧拓智能机器有限公司 一种基于图像和激光雷达融合的目标测距系统及方法
CN111985322B (zh) * 2020-07-14 2024-02-06 西安理工大学 一种基于激光雷达的道路环境要素感知方法
CN111882610B (zh) * 2020-07-15 2022-09-20 中国科学院自动化研究所 基于椭圆锥人工势场的服务机器人抓取目标物体的方法
CN111932688B (zh) * 2020-09-10 2021-02-09 深圳大学 一种基于三维点云的室内平面要素提取方法、系统及设备
CN112184736B (zh) * 2020-10-10 2022-11-11 南开大学 一种基于欧式聚类的多平面提取方法
CN112365575B (zh) * 2020-11-10 2022-06-21 广州极飞科技股份有限公司 地平面数据测量方法、装置、可移动设备和可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110458805A (zh) * 2019-03-26 2019-11-15 华为技术有限公司 一种平面检测方法、计算设备以及电路系统
US20200402300A1 (en) * 2019-06-21 2020-12-24 Harbin Institute Of Technology Terrain modeling method that fuses geometric characteristics and mechanical charateristics, computer readable storage medium, and terrain modeling system thereof
CN111639682A (zh) * 2020-05-13 2020-09-08 北京三快在线科技有限公司 基于点云数据的地面分割方法及装置
CN112329946A (zh) * 2020-11-10 2021-02-05 广州极飞科技有限公司 地面分割学习模型训练方法、地平面确定方法及相关装置
CN113240678A (zh) * 2021-05-10 2021-08-10 青岛小鸟看看科技有限公司 平面信息检测方法及系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117635888A (zh) * 2023-12-07 2024-03-01 腾讯科技（深圳）有限公司 一种数据处理方法及相关装置
CN117635888B (zh) * 2023-12-07 2024-04-26 腾讯科技（深圳）有限公司 一种数据处理方法及相关装置

Also Published As

Publication number Publication date
CN113240678B (zh) 2023-05-30
CN113240678A (zh) 2021-08-10

Similar Documents

Publication Publication Date Title
US10977818B2 (en) Machine learning based model localization system
WO2022237026A1 (zh) 平面信息检测方法及系统
US10198823B1 (en) Segmentation of object image data from background image data
CN108895981B (zh) 一种三维测量方法、装置、服务器和存储介质
WO2020206903A1 (zh) 影像匹配方法、装置及计算机可读存储介质
WO2018107910A1 (zh) 一种全景视频融合方法及装置
Gao et al. Robust RGB-D simultaneous localization and mapping using planar point features
JP5722502B2 (ja) モバイルデバイスのための平面マッピングおよびトラッキング
CN104380338B (zh) 信息处理器以及信息处理方法
WO2020063139A1 (zh) 脸部建模方法、装置、电子设备和计算机可读介质
EP3467788B1 (en) Three-dimensional model generation system, three-dimensional model generation method, and program
EP3467787A1 (en) Three-dimensional model generation system, three-dimensional model generation method, and program
JP2011095797A (ja) 画像処理装置、画像処理方法及びプログラム
US20190304112A1 (en) Methods and systems for providing selective disparity refinement
CN112712584B (zh) 空间建模方法、装置、设备
JP2013524593A (ja) 複数カメラのキャリブレーション用の方法および構成
WO2023071790A1 (zh) 目标对象的姿态检测方法、装置、设备及存储介质
CN110998671B (zh) 三维重建方法、装置、系统和存储介质
CN108628442B (zh) 一种信息提示方法、装置以及电子设备
WO2023024441A1 (zh) 模型重建方法及相关装置、电子设备和存储介质
CN106264537B (zh) 图像中人体姿态高度的测量系统及方法
US20230334727A1 (en) 2d and 3d floor plan generation
CN108492284B (zh) 用于确定图像的透视形状的方法和装置
Jung et al. Object Detection and Tracking‐Based Camera Calibration for Normalized Human Height Estimation
CN113298708A (zh) 三维房型的生成方法、装置及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21941593

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21941593

Country of ref document: EP

Kind code of ref document: A1