CN110458805A

CN110458805A - Plane detection method, computing device and circuit system

Info

Publication number: CN110458805A
Application number: CN201910605510.4A
Authority: CN
Inventors: 何凯文; 李阳; 刘昆
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-03-26
Filing date: 2019-07-05
Publication date: 2019-11-15
Anticipated expiration: 2039-07-05
Also published as: CN110458805B

Abstract

The application discloses a plane detection method, a computing device and a circuit system, wherein the method comprises the following steps: the computing equipment acquires image data to be processed, and then the image data to be processed is segmented to obtain N sub-image data, wherein N is an integer larger than 1. And then, determining point cloud information corresponding to at least one sub-image data in the N sub-image data, clustering point clouds corresponding to the N sub-image data according to the point cloud information corresponding to at least one sub-image data in the N sub-image data to obtain K crude extraction planes, and optimizing the K crude extraction planes to obtain L optimized planes, wherein K is a positive integer not greater than N, and L is a positive integer not greater than K. This may enable the computing device to detect more than one plane in the image data.

Description

A kind of plane monitoring-network method calculates equipment and circuit system

This application claims on 03 26th, 2019 submission Patent Office of the People's Republic of China, application No. is 201910234537.7, application The priority of the Chinese patent application of entitled " a kind of plane monitoring-network method and electronic equipment ", entire contents pass through reference knot It closes in this application.

Technical field

This application involves technical field of computer vision more particularly to a kind of plane monitoring-network method, calculate equipment and electricity Road system.

Background technique

Currently, object dimensional based on mobile phone rebuild and augmented reality (augmented reality, AR) game at For reality, and by the favor of major cell phone manufacturer and users.The meter with augmented reality game is rebuild in object dimensional During calculation, three-dimensional space plane monitoring-network is an important and basic function, because after detecting plane, ability The anchor point of object is further determined that, to render object at the anchor point determined.Currently, three-dimensional planar detection function is It is injected towards in various small devices, but the computing capability of these equipment is limited, can not handle the higher algorithm of complexity.

Modern AR application program, such as AR game, object, personage's modeling program, interactive process such as based on complex scene, Mixed reality application etc. proposes very high requirement to the scene understanding ability of rear end program, and one of them are namely based on image biography The three-dimensional structure of sensor or depth transducer comprehension of information scene.From the point of view of current mobile terminal computing capability, current back end journey Ordered pair scene understands that the processing capacity of algorithm is very limited.Before this, the dominant plane structure in scene can only be all determined, The position of maximum plane in three dimensions i.e. in scene.Commercial AR solution is also substantially based in scene now Principal plane (i.e. single plane) carry out the work in later period, including model, render.

As favorable rating of the consumer to artificial intelligence application increasingly increases, demand of the user to function of application It gradually increases, requirement of the augmented reality application program to rear end scene understanding ability also increases accordingly, and single plane monitoring-network is Through being unable to satisfy the use demand of user.

Summary of the invention

The application provides a kind of plane monitoring-network method, calculates equipment and circuit system, use so that calculate equipment can Detect the more than one plane in image data.

In a first aspect, the embodiment of the present application provides a kind of plane monitoring-network method, this method is applied to calculate equipment, the party Method includes: to calculate equipment to obtain image data to be processed, is then split to image data to be processed, obtains N number of subgraph Data, wherein N is the integer greater than 1.Later, it calculates equipment and determines at least one sub-image data in N number of sub-image data Corresponding point cloud information, then according to the corresponding point cloud information of at least one sub-image data in N number of sub-image data, to N Corresponding cloud of a sub-image data carries out clustering processing, obtains K coarse extraction plane, optimizes to K coarse extraction plane Processing obtains plane after L optimization, and wherein K is the positive integer no more than N, and L is the positive integer no more than K.

In the embodiment of the present application, by being split to image data to be processed, N number of sub-image data is obtained, later to N Corresponding cloud of a sub-image data carries out clustering processing, compared to the prior art in from center to begin stepping through image to be processed straight To detecting that a plane just stops detecting, it is only able to detect the scheme of a plane, scheme provided by the present application can be treated N number of sub-image data that processing image segmentation obtains carries out clustering processing, and calculating equipment can be made to be able to detect that figure in this way As the more than one plane in data.

In a kind of possible design, which is depth image, includes each picture in the depth image The image coordinate and depth value of vegetarian refreshments.It calculates equipment and gets depth image, depth image is split later, is obtained N number of Then sub- depth image calculates equipment and determines corresponding cloud letter of the sub- depth image of at least one of N number of sub- depth image Breath, and according to the corresponding point cloud information of each sub-image data in N number of sub- depth image, determine at least one sub- depth image The mean square error of plane after the fitting of corresponding cloud.Later, it calculates equipment and determines satisfaction first from N number of sub- depth image The sub- depth image of condition forms subgraph image set to be processed, concentrates the sub- depth image for including corresponding subgraph to be processed Point cloud is clustered, and K coarse extraction plane is obtained.Wherein, first condition include corresponding cloud of sub- depth image fitting after The mean square error of plane is less than or equal to first threshold.

In this design, by being split to depth image, N number of sub- depth image is obtained, then to N number of sub- depth map Corresponding cloud of the sub- depth image for meeting first condition as in carries out clustering processing, and meets the sub- depth map of first condition As corresponding cloud fitting after the mean square error of plane be less than or equal to first threshold, the son for illustrating to meet the first condition is deep Point in degree corresponding cloud of image is in one plane, that is to say, that may be to corresponding cloud in N number of sub- depth image The sub- depth image of plane, which screens, to be clustered, for cloud be not a plane sub- depth image without cluster, So not needing to cluster all sub- depth images in N number of sub- depth image, to not only can detecte sub- depth More than one plane in image, and the time of plane monitoring-network process can be saved.

In a kind of possible design, image data to be processed is depth image.It calculates equipment and obtains depth image, to depth Degree image is split, and obtains N number of sub- depth image.Then the sub- depth map of at least one of N number of sub- depth image is determined As corresponding point cloud information, using every corresponding cloud of sub- depth image in N number of sub- depth image as a node, structure figures Structure, each node in the graph structure are stored with the corresponding point cloud information of the node.It calculates every in equipment traversal graph structure A node determines two nodes for meeting second condition in graph structure, then between two nodes for meeting second condition Construct side；Wherein, second condition includes that the depth value of corresponding cloud of any node in two nodes is continuous and two nodes pair The angle that should be put between the normal vector of cloud is less than angle threshold value.Later, equipment is calculated to determine to tie in N number of sub- depth image with figure With the corresponding sub- depth image of node at least one side in structure, subgraph image set to be processed is formed, to subgraph collection to be processed In include corresponding cloud of sub- depth image clustered, obtain K coarse extraction plane.

In this design, by being split to depth image, N number of sub- depth image is obtained, then according to N number of sub- depth The corresponding point cloud information of image constructs graph structure, constructs side between two nodes for meeting second condition in graph structure later, That is, constructing a side between two nodes that may be fitted to plane, carried out later for the node for having side subsequent Clustering processing, so there is no need to carry out subsequent processing for sub- depth image lower a possibility that being fitted to plane, from And the time of entire detection planarization process can be saved.

In a kind of possible design, the image data to be processed include by binocular camera shooting the first RGB image and Second RGB image.After calculating equipment gets the first RGB image and the second RGB image, first RGB image is carried out Image pre-segmentation obtains N number of first face block, and carries out image pre-segmentation to second RGB image, obtains N number of second face block, Wherein the first face block and the second face block have position corresponding relationship.For each first face block in the block of N number of first face, equipment is calculated Execute: determined from the block of N number of second face with the first face block have position corresponding relationship the second face block, according to the first face block, And there is the second face block of position corresponding relationship with the first face block, it determines disparity map, sub- depth is determined according to disparity map Image.It may thereby determine that from N number of sub- depth image.Then, calculating equipment can be according to the N number of sub- depth image group determined At subgraph image set to be processed, the corresponding point cloud information of the sub- depth image of at least one of N number of sub- depth image is determined, it Afterwards, equipment is calculated according to the corresponding point cloud information of at least one sub-image data in N number of sub-image data, to subgraph to be processed The N number of corresponding cloud of sub- depth image for including in image set is clustered, and K coarse extraction plane is obtained.

In this design, it is split respectively by the first RGB image and the second RGB image that are shot to binocular camera, It obtains to determine N number of disparity map for the N number of first face block and N number of second face block of plane, then can determine N number of possibility For the sub- depth image of plane.Then it can may be clustered to N number of for corresponding cloud of sub- depth image of plane, without It needs to carry out clustering processing to the whole image in the first RGB image and the second RGB image, so as to reduce treating capacity, such as This, can not only detect the more than one plane in image data, and can accelerate to detect the speed of plane.

In a kind of possible design, a kind of sub- depth that may be implemented to include to subgraph to be processed concentration presented below Corresponding cloud of image is clustered, and the mode of K coarse extraction plane is obtained, in this approach, calculate equipment can according to Processing subgraph concentrates the corresponding point cloud information of the sub- depth image of every for including, and establishes minimum heap data structure, wherein minimum The son that heap data structure is used to concentrate subgraph to be processed according to the mean square error of every sub- corresponding cloud of depth image is deep Degree image is ranked up, and the mean square error for being located at the corresponding cloud of sub- depth image on heap top is minimum.For minimum heap data Structure executes predetermined registration operation, until plane after the fitting of the corresponding cloud of any two node in minimum heap data structure Mean square error is greater than first threshold, obtains K coarse extraction plane；Wherein, predetermined registration operation includes: the heap from minimum heap data structure Sub- depth image is taken out in top, the son for meeting third condition is deep if determining from the sub- depth image adjacent with sub- depth image Image is spent, then is merged sub- depth image with the sub- depth image for meeting third condition, sub- depth image after being merged； Third condition include mean square error after point-cloud fitting plane corresponding with the sub- depth image be less than first threshold and it is described Square error is minimum；Depth image after fusion is added into minimum heap data structure.

In this design, it may be implemented by establishing minimum heap data structure to N number of corresponding cloud of sub- depth image Mean square error is ranked up, and the mean square error of the corresponding cloud of sub- depth image at the heap bottom of most rickle to heap top is smaller and smaller, Mean square error positioned at the corresponding cloud of sub- depth image on heap top is minimum, so as to take out from the heap top of most rickle every time The smallest corresponding cloud of sub- depth image of mean square error, preferentially son the smallest to the mean square error in minimum heap data structure is deep Degree corresponding cloud of image is clustered, i.e., preferentially carries out clustering processing to the most possible node for forming a plane, in this way Can as fast as possible finding may be the node of a plane, and merged.For each from minimum heap data structure Corresponding cloud of the sub- depth image taken out, if another corresponding cloud of sub- depth image that can be merged can be found, So fused cloud is continued to be pressed into most rickle and be ranked up, until each sub- depth image pair remaining in most rickle It cannot continue to merge between the point cloud answered, end of clustering.So as to accelerate to determine the speed of coarse extraction plane.

In a kind of possible design, which is the point cloud for including in three-dimensional space.To described wait locate Reason image data is split, and is obtained N number of sub-image data, be may include: using the three-dimensional space as octree structure The node of first level；Using three-dimensional space as the node of the first level of octree structure, octree structure is constructed；For eight Each child node that the first level and the i-th level in fork tree construction include, calculating equipment can perform the following operations: if should Child node meets fourth condition, then carries out eight equal portions segmentations to the child node, obtain eight child nodes of i+1 level；Until All child nodes for including in last level meet fifth condition, building obtain include the child node of M level Octree Structure；Wherein fourth condition includes that the mean square error of corresponding cloud of child node is greater than first threshold；I is the integer greater than 1, the I level includes 8i child node, and fifth condition includes the mean square error of corresponding cloud of child node no more than first threshold, or Person, corresponding cloud of child node include that point quantity is less than amount threshold.Then, it calculates equipment and determines N number of undivided child node At least one of the undivided corresponding point cloud information of child node.Later, equipment is calculated according in N number of undivided child node The corresponding point cloud information of at least one undivided child node, it is corresponding to undivided child node N number of in octree structure Point cloud carries out clustering processing, obtains K coarse extraction plane.

In this design, using the point cloud in three-dimensional space as the first-level nodes of octree structure, and to the first layer Node is split as eight equal portions, later, for each child node in each level, the mean square error of corresponding cloud of child node Greater than first threshold, illustrate point that corresponding cloud of the child node includes not in one plane, so, to the mean square error of cloud The child node that difference is greater than first threshold is split, until the point that corresponding cloud of each child node in octree structure includes In one plane, alternatively, corresponding cloud of child node point for including is few enough, that is, the quantity put is less than amount threshold, no longer Continue to divide, include two classes in the N number of child node for not continuing segmentation in octree structure finally obtained in this way: one kind is point The mean square error of cloud is less than or equal to the child node of first threshold, and the another kind of point quantity for including for point cloud is less than amount threshold Child node.Later, clustering processing, available more than one plane are carried out to the N number of child node for not continuing segmentation.

In a kind of possible design, according to the undivided child node pair of at least one of N number of undivided child node The point cloud information answered carries out clustering processing to corresponding cloud of undivided child node N number of in octree structure, obtains K slightly Plane is extracted, may include: to calculate equipment to be corresponded to according to the undivided child node of at least one of N number of undivided child node Point cloud information, determine the normal vector of each corresponding cloud of undivided child node in N number of undivided child node, and by method Vector passes through Hough transformation at the point in parameter space, then, it is determined that the method for N number of corresponding cloud of undivided child node out The K point set that vector is formed in parameter space, each point set have an aggregation center；For each point set, determination is grown Point in the pericentral preset range of aggregation of the point set will fall the corresponding undivided son section of point within a preset range Corresponding cloud of point permeates coarse extraction plane.It should be noted that undivided child node herein refers to leaf Node, the i.e. undivided child node do not have child node.

In this design, the normal vector of N number of corresponding cloud of undivided child node is turned by Hough (Hough) transformation As soon as the point in parameter space is changed into, then the point that the normal vector for belonging to conplane cloud is formed in parameter space It can flock together, these points will tend to a central point, so determining the point set formed in parameter space, join The corresponding coarse extraction plane of each point set is formed in number space, can be quickly obtained N number of undivided child node tool in this way There is the child node of coplanar relation, the child node with coplanar relation is then merged into a coarse extraction plane, may be implemented quickly Obtain more than one coarse extraction plane.

In a kind of possible design, one kind presented below may be implemented to optimize processing to K coarse extraction plane, obtain The mode of plane after to L optimization: it determines the normal vector of each coarse extraction plane in K coarse extraction plane, traverses K coarse extraction Any coarse extraction plane in plane, meets the coarse extraction plane of Article 6 part if it exists, then by coarse extraction plane and meets the The coarse extraction plane of six conditions permeates plane, obtain L optimize after plane.Wherein, Article 6 part include: normal vector with The normal vector of coarse extraction plane is parallel and is less than variance threshold values with the variance after coarse extraction plane fitting plane.

By the design, the multiple coarse extraction planes in K coarse extraction plane being actually a plane can be carried out Fusion, obtained from more accurately L optimize after plane.

Second aspect, the embodiment of the present application provide a kind of plane monitoring-network method, and this method is applied to calculate equipment, the party Method includes: to obtain image data to be processed；Semantic segmentation is carried out to image data to be processed, is obtained N number of with markup information Sub-image data, N are the integer greater than 1；The markup information is used to mark the target object in sub-image data；According to every height The markup information of image data determines the Q subgraph numbers with plane from N number of sub-image data with markup information According to；Q is the integer greater than 0 and less than or equal to N；Determine that each there is plane in the Q sub-image datas with plane The corresponding point cloud information of sub-image data；According to the subgraph number each in the Q sub-image datas with plane with plane According to corresponding point cloud information, K coarse extraction plane is determined from the Q sub-image datas with plane；K be more than or equal to The integer of Q；Processing is optimized to K coarse extraction plane, obtains plane after L optimization；L is the positive integer no more than K.

Based on the program, semantic segmentation, available N number of subgraph with markup information are carried out to data image to be processed As data, the Q sub-image datas with plane are then determined from N number of sub-image data with markup information, thus It can only need to carry out plane monitoring-network to the Q sub-image datas with plane, not need to the subgraph number for not having plane According to plane monitoring-network is carried out, so as to reduce treating capacity, and the Q sub-image datas with plane are handled, it can be with Detect more than one plane.

In a kind of possible design, the image data to be processed include by binocular camera shooting the first RGB image and Second RGB image.Semantic segmentation is carried out to image data to be processed, obtains N number of sub-image data with markup information, it can be with Include: that semantic segmentation is carried out to the first RGB image, obtains N number of the first subgraph with markup information, and scheme to the 2nd RGB As carrying out semantic segmentation, N number of the second subgraph with markup information is obtained, wherein each the first son with markup information Image and the second subgraph with the first subgraph with position corresponding relationship with markup information, which form one, has mark Infuse the sub-image data of information.It determines in the Q sub-image datas with plane each with the sub-image data pair of plane The point cloud information answered may include: for the subgraph number of the Q sub-image datas with plane with plane each of According to calculating equipment can be performed following operation: according to first with markup information for including in the sub-image data with plane Subgraph and the second subgraph with markup information with the first subgraph with position corresponding relationship, determine parallax Figure, determines sub- depth image according to disparity map, according to sub- depth image, determines to have the sub-image data of plane corresponding Point cloud information.According to the sub-image data corresponding point cloud information each in Q sub-image datas with plane with plane, K coarse extraction plane is determined from the Q sub-image datas with plane, may include: according to the subgraph with plane The corresponding point cloud information of data determines K coarse extraction plane from Q sub- depth images.

In this design, in the first RGB image and the 2nd RGB figure that image data to be processed includes by binocular camera shooting When picture, semantic segmentation, then N number of first subgraph obtained from segmentation can be carried out to the first RGB image and the second RGB image respectively Respectively obtained in picture and N number of second subgraph first subgraph of the Q with plane and with first subgraph with plane Q with position corresponding relationship the second subgraphs with plane, further according to Q the first subgraphs and Q with plane The second subgraph with plane obtains Q disparity map, then obtains Q sub- depth images according to Q disparity map, so as to Only need to carry out plane monitoring-networks with the sub- depth images of plane to Q, do not need to the sub-image data for not having plane into Row plane monitoring-network, so as to reduce treating capacity.

In a kind of possible design, according to the corresponding point cloud information of sub-image data with plane, from Q sub- depth K coarse extraction plane is determined in image, may include: to make corresponding cloud of every sub- depth image in Q sub- depth images For a node, graph structure is constructed, wherein each node in graph structure is stored with the corresponding point cloud information of node.Time then, Each node in graph structure is gone through, two nodes for meeting second condition in graph structure is determined, is then meeting second condition Two nodes between construct side, wherein second condition include corresponding cloud of any node in two nodes depth value connect Angle between the continuous and normal vector of two node corresponding points clouds is less than angle threshold value；Determine in Q sub- depth images with figure With the corresponding sub- depth image of node at least one side in structure, subgraph image set to be processed is formed；To subgraph to be processed Corresponding cloud of sub- depth image that concentration includes carries out clustering processing, obtains K coarse extraction plane.

In this design, graph structure is constructed according to the corresponding point cloud information of the sub- depth image of Q, later to full in graph structure Side is constructed between two nodes of sufficient second condition, that is to say, that constructs one between two nodes that may be fitted to plane A side carries out subsequent clustering processing for the node for having side later, that is to say, that the sub- depth map that may be fitted to plane As carrying out clustering processing, so there is no need to carry out subsequent place for sub- depth image lower a possibility that being fitted to plane Reason, so as to save the time of entire detection planarization process.

In a kind of possible design, the corresponding cloud of sub- depth image for including is concentrated to gather subgraph to be processed Class obtains K coarse extraction plane, may include: to be concentrated every sub- depth image for including corresponding according to subgraph to be processed Point cloud information establishes minimum heap data structure, wherein minimum heap data structure is used for according to the corresponding point of every sub- depth image The mean square error of cloud is ranked up the sub- depth image that subgraph to be processed is concentrated, and the sub- depth image for being located at heap top is corresponding Point cloud mean square error it is minimum.Then, for minimum heap data structure, predetermined registration operation is executed, until minimum heap data structure In corresponding cloud of any two node fitting after plane mean square error be greater than first threshold, obtain K coarse extraction put down Face；Wherein, predetermined registration operation include: from taking out sub- depth image in heap top in minimum heap data structure, if from sub- depth image The sub- depth image for meeting third condition is determined in adjacent sub- depth image, then sub- depth image and will meet third condition Sub- depth image merged, sub- depth image after being merged；Third condition includes point corresponding with the sub- depth image Mean square error after cloud fit Plane is less than first threshold and the mean square error is minimum；Sub- depth image is added after merging Into minimum heap data structure.

In this design, it may be implemented to concentrate the Q for including to subgraph to be processed by establishing minimum heap data structure The mean square error of sub- corresponding cloud of depth image is ranked up, the corresponding point of sub- depth image on heap bottom to the heap top of most rickle The mean square error of cloud is smaller and smaller, and the mean square error positioned at the corresponding cloud of sub- depth image on heap top is minimum, so as to every The smallest corresponding cloud of sub- depth image of mean square error is taken out in the secondary heap top from most rickle, preferentially to minimum heap data structure In the smallest corresponding cloud of sub- depth image of mean square error clustered, i.e., preferentially form plane to most possible Node carries out clustering processing, in this way can as fast as possible finding may be the node of a plane, and merged.For every A corresponding cloud of the sub- depth image taken out from minimum heap data structure, if another son that can be merged can be found Corresponding cloud of depth image is ranked up then fused cloud is continued to be pressed into most rickle, until remaining in most rickle It cannot continue to merge between the corresponding cloud of sub- depth image of remaining each, end of clustering.So as to accelerate to determine slightly to mention It makes even the speed in face.

In a kind of possible design, processing is optimized to K coarse extraction plane, obtains plane after L optimization, it can be with Comprise determining that the normal vector of each coarse extraction plane in K coarse extraction plane, traverse in K coarse extraction plane any slightly mentions It makes even face, meets the coarse extraction plane of Article 6 part if it exists, then put down coarse extraction plane with the coarse extraction for meeting Article 6 part Face permeates plane, obtain L optimize after plane.Wherein, Article 6 part includes: the normal direction of normal vector Yu coarse extraction plane Amount is parallel and is less than variance threshold values with the variance after coarse extraction plane fitting plane.

The third aspect, the embodiment of the present application provide a kind of calculating equipment, including at least one processor.This at least one Processor is configured as performing the following operations: obtaining image data to be processed；Image data to be processed is split, N is obtained A sub-image data, N are the integer greater than 1；Determine corresponding cloud of at least one sub-image data in N number of sub-image data Information；According to the corresponding point cloud information of at least one sub-image data in N number of sub-image data, to N number of sub-image data pair The point cloud answered carries out clustering processing, obtains K coarse extraction plane；K is the positive integer no more than N；K coarse extraction plane is carried out Optimization processing obtains plane after L optimization；L is the positive integer no more than K.

In a kind of possible design, which is depth image；It include each pixel in depth image The image coordinate and depth value of point；Image data to be processed is split, obtains N number of sub-image data, comprising: to depth map As being split, N number of sub- depth image is obtained；Determine the corresponding point of at least one sub-image data in N number of sub-image data Cloud information, comprising: determine the corresponding point cloud information of the sub- depth image of at least one of N number of sub- depth image；According to N number of son The corresponding point cloud information of at least one sub-image data in image data gathers corresponding cloud of N number of sub-image data Class processing, obtains K coarse extraction plane, comprising: according to corresponding cloud letter of each sub-image data in N number of sub- depth image Breath determines the mean square error of plane after the fitting of every sub- corresponding cloud of depth image；It is determined from N number of sub- depth image The sub- depth image for meeting first condition forms subgraph image set to be processed；First condition is corresponding cloud of sub- depth image The mean square error of plane is less than or equal to first threshold after fitting；Concentrate the sub- depth image for including corresponding subgraph to be processed Point cloud clustered, obtain K coarse extraction plane.

In a kind of possible design, image data to be processed is depth image.Image data to be processed is split, N number of sub-image data is obtained, may include: to be split to depth image, obtains N number of sub- depth map.Determine N number of subgraph number The corresponding point cloud information of at least one sub-image data in may include: at least one determined in N number of sub- depth image The corresponding point cloud information of a sub- depth image.According to corresponding cloud of at least one sub-image data in N number of sub-image data Information carries out clustering processing to corresponding cloud of N number of sub-image data, obtains K coarse extraction plane, may include: by N number of son Every corresponding cloud of sub- depth image constructs graph structure as a node in depth image；Each node in graph structure It is stored with the corresponding point cloud information of node；Each node in graph structure is traversed, determines to meet second condition in graph structure Two nodes, then construct side between two nodes for meeting second condition；Second condition includes any section in two nodes Angle of the depth value of corresponding cloud of point continuously and between the normal vector of two node corresponding points clouds is less than angle threshold value；Really Sub- depth image corresponding with having the node at least one side in graph structure in N number of sub- depth image is made, is formed to be processed Subgraph image set；It concentrates the corresponding cloud of sub- depth image for including to cluster subgraph to be processed, it is flat to obtain K coarse extraction Face.

In a kind of possible design, image data to be processed includes by the first RGB image and the of binocular camera shooting Two RGB images；Image data to be processed is split, N number of sub-image data is obtained, may include: to the first RGB image into Row image pre-segmentation obtains N number of first face block；And image pre-segmentation is carried out to the second RGB image, obtain N number of second face block；The Block and the second face block have position corresponding relationship on one side；For each first face block in the block of N number of first face, calculating equipment can be held The following operation of row: the second face block that there is position corresponding relationship with the first face block is determined from the block of N number of second face；According to first Face block and the second face block with the first face block with position corresponding relationship, determine disparity map；Son is determined according to disparity map Depth image；Subgraph image set to be processed is formed according to the N number of sub- depth image determined.It determines in N number of sub-image data extremely Few corresponding point cloud information of a sub-image data, may include: to determine the sub- depth of at least one of N number of sub- depth image The corresponding point cloud information of image；According to the corresponding point cloud information of at least one sub-image data in N number of sub-image data, to N Corresponding cloud of a sub-image data carries out clustering processing, obtains K coarse extraction plane, may include: according to N number of subgraph number The corresponding point cloud information of at least one sub-image data in concentrates the N number of sub- depth image for including to subgraph to be processed Corresponding cloud is clustered, and K coarse extraction plane is obtained.

In a kind of possible design, the corresponding cloud of sub- depth image for including is concentrated to gather subgraph to be processed Class obtains K coarse extraction plane, may include: to be concentrated every sub- depth image for including corresponding according to subgraph to be processed Point cloud information establishes minimum heap data structure；Minimum heap data structure is used for according to corresponding cloud of every sub- depth image Mean square error is ranked up the sub- depth image that subgraph to be processed is concentrated, and is located at the corresponding point of sub- depth image on heap top The mean square error of cloud is minimum.For minimum heap data structure, predetermined registration operation is executed, until any two in minimum heap data structure The mean square error of plane is greater than first threshold after the fitting of a corresponding cloud of node, obtains K coarse extraction plane；Wherein, in advance If operation includes: from taking out sub- depth image in heap top in minimum heap data structure, if deep from the son adjacent with sub- depth image The sub- depth image for meeting third condition is determined in degree image, then by sub- depth image and the sub- depth map for meeting third condition As being merged, sub- depth image after being merged；Third condition includes point-cloud fitting plane corresponding with the sub- depth image Mean square error afterwards is less than first threshold and the mean square error is minimum；Depth image after fusion is added to most rickle number According in structure.

In a kind of possible design, image data to be processed is the point cloud for including in three-dimensional space.To image to be processed Data are split, and obtain N number of sub-image data, may include: using three-dimensional space as the first level of octree structure Node；For the first level and the i-th level each child node for including in octree structure, calculate equipment can execute with Lower operation: if child node meets fourth condition, carrying out eight equal portions segmentations to child node, obtains eight son sections of i+1 level Point；Wherein fourth condition includes that the mean square error of corresponding cloud of child node is greater than first threshold；I is integer greater than 1, i-th Level includes 8i child node；Until all child nodes for including in last level meet fifth condition, building obtains including M The octree structure of the child node of a level；Wherein, fifth condition includes that the mean square error of corresponding cloud of child node is not more than First threshold, alternatively, corresponding cloud of child node includes that point quantity is less than amount threshold；Determine in octree structure it is N number of not The child node of segmentation.Determine the corresponding point cloud information of at least one sub-image data in N number of sub-image data；Determine N number of son The corresponding point cloud information of at least one sub-image data in image data may include: in determining N number of undivided child node The corresponding point cloud information of at least one undivided child node.According at least one subgraph number in N number of sub-image data According to corresponding point cloud information, clustering processing is carried out to corresponding cloud of N number of sub-image data, obtains K coarse extraction plane, it can be with It include: the child node corresponding point cloud information undivided according at least one of N number of undivided child node, to Octree knot Corresponding cloud of N number of undivided child node carries out clustering processing in structure, obtains K coarse extraction plane.

In a kind of possible design, according to the undivided child node pair of at least one of N number of undivided child node The point cloud information answered carries out clustering processing to corresponding cloud of undivided child node N number of in octree structure, obtains K slightly Plane is extracted, may include: child node corresponding cloud letter undivided according at least one of N number of undivided child node Breath, determines the normal vector of each corresponding cloud of undivided child node in N number of undivided child node, and normal vector is passed through Hough transformation is at the point in parameter space；Determine the normal vector of N number of corresponding cloud of undivided child node in parameter sky Between middle formation K point set, each point set have an aggregation center；For each point set, the aggregation for falling in point set is determined Point in pericentral preset range；The corresponding point Yun Rong of the corresponding undivided child node of point within a preset range will be fallen It is combined into a coarse extraction plane.

In a kind of possible design, processing is optimized to K coarse extraction plane, obtains plane after L optimization, it can be with Comprise determining that the normal vector of each coarse extraction plane in K coarse extraction plane；Any slightly mentioning in K coarse extraction plane of traversal It makes even face, meets the coarse extraction plane of Article 6 part if it exists, then put down coarse extraction plane with the coarse extraction for meeting Article 6 part Face permeates plane, obtain L optimize after plane；Wherein, Article 6 part includes: the normal direction of normal vector Yu coarse extraction plane Amount is parallel and is less than variance threshold values with the variance after coarse extraction plane fitting plane.

Fourth aspect, the embodiment of the present application provide a kind of calculating equipment, including at least one processor.This at least one Processor is configured as performing the following operations: obtaining image data to be processed；Semantic segmentation is carried out to image data to be processed, N number of sub-image data with markup information is obtained, N is the integer greater than 1；Markup information is for marking in sub-image data Target object；According to the markup information of each sub-image data, Q is determined from N number of sub-image data with markup information A sub-image data with plane；Q is the integer greater than 0 and less than or equal to N；Determine the Q subgraphs with plane Each with the corresponding point cloud information of sub-image data of plane in data；According to every in the Q sub-image datas with plane The corresponding point cloud information of a sub-image data with plane determines that K slightly mention from the Q sub-image datas with plane It makes even face；K is the integer more than or equal to Q；Processing is optimized to K coarse extraction plane, obtains plane after L optimization；L is Positive integer no more than K.

In a kind of possible design, image data to be processed includes by the first RGB image and the of binocular camera shooting Two RGB images；Semantic segmentation is carried out to image data to be processed, obtains N number of sub-image data with markup information, N is big In 1 integer, it may include: that semantic segmentation is carried out to the first RGB image, obtain N number of the first subgraph with markup information； And semantic segmentation is carried out to the second RGB image, obtain N number of the second subgraph with markup information；Wherein, each that there is mark First subgraph of information and the second subgraph group with markup information with the first subgraph with position corresponding relationship At a sub-image data with markup information；Determine that each there is plane in the Q sub-image datas with plane The corresponding point cloud information of sub-image data may include: to have plane for each of Q sub-image datas with plane Sub-image data, calculate equipment can execute following operation: according to include in the sub-image data with plane have mark Infuse the first subgraph of information and the second subgraph with markup information with the first subgraph with position corresponding relationship Picture determines disparity map；Sub- depth image is determined according to disparity map；According to sub- depth image, the son with plane is determined The corresponding point cloud information of image data；According to the sub-image data each in the Q sub-image datas with plane with plane Corresponding point cloud information determines K coarse extraction plane from the Q sub-image datas with plane, may include: according to tool There is the corresponding point cloud information of the sub-image data of plane, determines K coarse extraction plane from Q sub- depth images.

In a kind of possible design, according to the corresponding point cloud information of sub-image data with plane, from Q sub- depth K coarse extraction plane is determined in image, may include: to make corresponding cloud of every sub- depth image in Q sub- depth images For a node, graph structure is constructed；Each node in graph structure is stored with the corresponding point cloud information of node；It traverses in graph structure Each node, determine two nodes for meeting second condition in graph structure, then in two nodes for meeting second condition Between construct side, wherein second condition includes that the depth value of corresponding cloud of any node in two nodes is continuous and two sections Angle between the normal vector of point corresponding points cloud is less than angle threshold value；Determine in Q sub- depth images with have in graph structure The corresponding sub- depth image of the node at least one side, forms subgraph image set to be processed；Include to subgraph to be processed concentration Corresponding cloud of sub- depth image carries out clustering processing, obtains K coarse extraction plane.

In a kind of possible design, the corresponding cloud of sub- depth image for including is concentrated to gather subgraph to be processed Class obtains K coarse extraction plane, may include: to be concentrated every sub- depth image for including corresponding according to subgraph to be processed Point cloud information establishes minimum heap data structure；Minimum heap data structure is used for according to corresponding cloud of every sub- depth image Mean square error is ranked up the sub- depth image that subgraph to be processed is concentrated, and is located at the corresponding point of sub- depth image on heap top The mean square error of cloud is minimum；For minimum heap data structure, predetermined registration operation is executed, until any two in minimum heap data structure The mean square error of plane is greater than first threshold after the fitting of a corresponding cloud of node, obtains K coarse extraction plane；Wherein, in advance If operation includes: from taking out sub- depth image in heap top in minimum heap data structure, if deep from the son adjacent with sub- depth image The sub- depth image for meeting third condition is determined in degree image, then by sub- depth image and the sub- depth map for meeting third condition As being merged, sub- depth image after being merged；Third condition includes point-cloud fitting plane corresponding with the sub- depth image Mean square error afterwards is less than first threshold and mean square error is minimum；Depth image after fusion is added to minimum heap data knot In structure.

In a kind of possible design, processing is optimized to K coarse extraction plane, obtains plane after L optimization, it can be with Comprise determining that the normal vector of each coarse extraction plane in K coarse extraction plane；Any slightly mentioning in K coarse extraction plane of traversal It makes even face, meets the coarse extraction plane of Article 6 part if it exists, then put down coarse extraction plane with the coarse extraction for meeting Article 6 part Face permeates plane；Wherein, Article 6 part includes: that normal vector is parallel with the normal vector of coarse extraction plane and and coarse extraction Variance after plane fitting plane is less than variance threshold values.

5th aspect, the embodiment of the present application also provide a kind of calculating equipment, which includes executing any of the above-described side Module/unit of the method for the possible design of any one of face.These module/units can be by hardware realization, can also be with Corresponding software realization is executed by hardware.

6th aspect, also provides a kind of computer readable storage medium in the embodiment of the present application, described computer-readable to deposit Storage media includes computer program, when computer program is run on an electronic device, so that electronic equipment execution is above-mentioned The method of the possible design of any one of either side.

7th aspect, the embodiment of the present application also provides a kind of program product, including instruction, when described program product is in electronics When being run in equipment, so that the method for calculating equipment and executing any one possible design of any of the above-described aspect.

In addition, third aspect technical effect brought by any possible design method into the 7th aspect can be found in first Technical effect brought by different designs mode in aspect or second aspect, details are not described herein again.

Detailed description of the invention

Fig. 1 is the structural schematic diagram of a kind of electronic equipment provided by the present application；

Fig. 2 is a kind of plane monitoring-network method flow schematic diagram provided by the present application；

Fig. 3 a is a kind of schematic diagram of a scenario provided by the present application；

Fig. 3 b is a kind of depth image schematic diagram provided by the present application；

Fig. 3 c~Fig. 3 h is plane monitoring-network process schematic provided by the present application；

Fig. 4 a is a kind of subdivision schematic diagram of three-dimensional space provided by the present application；

Fig. 4 b is a kind of octree structure schematic diagram provided by the present application；

Fig. 5 is another plane monitoring-network process schematic provided by the present application；

Fig. 6 is a kind of image schematic diagram including mark provided by the present application；

Fig. 7 is another image schematic diagram including mark provided by the present application；

Fig. 8 a is a kind of RGB image schematic diagram provided by the present application；

Fig. 8 b is the image schematic diagram of pre-segmentation provided by the present application；

Fig. 9 is another plane monitoring-network method flow schematic diagram provided by the present application；

Figure 10 is a kind of structural schematic diagram for calculating equipment provided by the present application.

Specific embodiment

This application provides a kind of plane monitoring-network method, equipment and circuit system are calculated, equipment is calculated and is based on getting Image data to be processed be split, and plane monitoring-network processing is carried out according to the obtained sub-image data of segmentation respectively, thus It can detecte multiple planes in scene, and then can be improved and calculate equipment to the understandability of scene.

In order to keep the purposes, technical schemes and advantages of the application clearer, below in conjunction with attached drawing to the application make into One step it is described in detail.

The scheme of plane monitoring-network provided by the embodiments of the present application can be applied to various calculating equipment, which can be Electronic equipment is also possible to server.Wherein electronic equipment can include but is not limited to personal computer, server computer, Hand-held or laptop devices, mobile device (such as mobile phone, mobile phone, tablet computer, personal digital assistant, media play Device etc.), consumer electronic devices, minicomputer, mainframe computer, mobile robot, unmanned plane etc..

When electronic equipment is needed to the plane in image data to be processed is detected, in a kind of possible implementation, electricity Sub- equipment can use plane monitoring-network method provided by the embodiments of the present application, realize that detecting multiple planes obtains testing result.Separately In a kind of possible implementation, image data to be processed can be sent to realizing plane monitoring-network process by electronic equipment Other equipment of processing capacity, such as server or terminal device, then the electronic equipment receives the inspection from other equipment Survey result.

In following embodiment, for calculating equipment and be electronic equipment, to the plane monitoring-network provided in the embodiment of the present application Method is introduced.

A kind of method for plane monitoring-network that present application example provides, suitable for electronic equipment as shown in Figure 1, below first letter Singly introduce the specific structure of electronic equipment.

Refering to what is shown in Fig. 1, the hardware structural diagram of the electronic equipment for the embodiment of the present application application.As shown in Figure 1, electric Sub- equipment 100 may include display equipment 110, processor 120 and memory 130.Wherein, memory 130 can be used for storing Software program and data, processor 120 can be stored in the software program and data of memory 130 by running, thus Execute the various function application and data processing of electronic equipment 100.

Memory 130 can mainly include storing program area and storage data area, wherein storing program area can store operation system Application program (such as image collecting function etc.) needed for system, at least one function etc.；Storage data area can be stored according to electronics Equipment 100 uses created data (such as audio data, text information, image data etc.) etc..In addition, memory 130 May include high-speed random access memory, can also include nonvolatile memory, a for example, at least disk memory, Flush memory device or other volatile solid-state parts.

Processor 120 is the control centre of electronic equipment 100, utilizes various interfaces and the entire electronic equipment of connection Various pieces execute electronic equipment 100 by running or executing the software program and/or data that are stored in memory 130 Various functions and processing data, to carry out integral monitoring to electronic equipment.Processor 120 may include one or more processing Unit, such as: processor 120 may include application processor (application processor, AP), modulation /demodulation processing Device, graphics processor (graphics processing unit, GPU), image-signal processor (image signal Processor, ISP), controller, memory, Video Codec, digital signal processor (digital signal Processor, DSP), baseband processor and/or neural network processor (Neural-network Processing Unit, NPU) etc..Wherein, different processing units can be independent device, also can integrate in one or more processors.

Wherein, NPU is neural network (neural-network, NN) computation processor, by using for reference biological neural network Structure, such as transfer mode between human brain neuron is used for reference, it, can also continuous self study to input information fast processing.It is logical Crossing NPU may be implemented the application such as intelligent cognition of electronic equipment 100, such as: image recognition, recognition of face, speech recognition, text Understand etc..

In some embodiments, processor 120 may include one or more interfaces.Interface may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit Sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiving-transmitting transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), universal input export (general-purpose Input/output, GPIO) interface, Subscriber Identity Module (subscriber identity module, SIM) interface, and/or Universal serial bus (universal serial bus, USB) interface etc..

I2C interface is a kind of bi-directional synchronization universal serial bus, including serial data line (serial data line, SDA) He Yigen serial time clock line (derail clock line, SCL).In some embodiments, processor 120 may include Multiple groups I2C bus.Processor 120 can distinguish coupled with touch sensors, charger, flash of light by different I2C bus interface Lamp, camera 160 etc..Such as: processor 120 can be made processor 120 and be touched by I2C interface coupled with touch sensors Sensor is communicated by I2C bus interface, realizes the touch function of electronic equipment 100.

I2S interface can be used for voice communication.In some embodiments, processor 120 may include multiple groups I2S bus. Processor 120 can be coupled by I2S bus with audio-frequency module, realize the communication between processor 120 and audio-frequency module.One In a little embodiments, audio-frequency module can transmit audio signal to WiFi module 190 by I2S interface, and realization passes through bluetooth headset The function of receiving calls.

Pcm interface can be used for voice communication, by analog signal sampling, quantization and coding.In some embodiments, sound Frequency module can be coupled with WiFi module 190 by pcm bus interface.In some embodiments, audio-frequency module can also pass through Pcm interface transmits audio signal to WiFi module 190, realizes the function of receiving calls by bluetooth headset.The I2S interface and The pcm interface may be used to voice communication.

UART interface is a kind of Universal Serial Bus, is used for asynchronous communication.The bus can be bidirectional communications bus. The data that it will be transmitted are converted between serial communication and parallel communications.In some embodiments, UART interface usually by with In connection processor 120 and WiFi module 190.Such as: processor 120 passes through the bluetooth in UART interface and WiFi module 190 Module communication, realizes Bluetooth function.In some embodiments, audio-frequency module can be passed by UART interface to WiFi module 190 Audio signal is passed, realizes the function of playing music by bluetooth headset.

MIPI interface can be used to connect the peripheral components such as processor 120 and display equipment 110, camera 160.MIPI Interface includes 160 serial line interface of camera (camera serial interface, CSI), display screen serial line interface (display Serial interface, DSI) etc..In some embodiments, processor 120 and camera 160 are communicated by CSI interface, real The shooting function of existing electronic equipment 100.Processor 120 and display screen realize the aobvious of electronic equipment 100 by DSI interface communication Show function.

GPIO interface can pass through software configuration.GPIO interface can be configured as control signal, may be alternatively configured as counting It is believed that number.In some embodiments, GPIO interface can be used for connecting processor 120 and camera 160, show equipment 110, WiFi module 190, audio-frequency module, sensor module etc..GPIO interface can be additionally configured to I2C interface, I2S interface, UART Interface, MIPI interface etc..

USB interface is the interface for meeting USB standard specification, specifically can be Mini USB interface, Micro USB interface, USB Type C interface etc..USB interface can be used for connecting charger for the charging of electronic equipment 100, can be used for electronics and sets Data are transmitted between standby 100 and peripheral equipment.It can be used for connection earphone, audio played by earphone.The interface can be with For connecting other electronic equipments, such as AR equipment etc..

It is understood that the interface connection relationship of each intermodule of signal of the embodiment of the present invention, only schematically illustrates, The structure qualification to electronic equipment 100 is not constituted.In other embodiments of the application, electronic equipment 100 can also be used The combination of different interface connection type or multiple interfaces connection type in above-described embodiment.

It further include the camera 160 for shooting image or video in electronic equipment 100.Camera 160 can be commonly Camera is also possible to focusing camera head.

Electronic equipment 100 can also include input equipment 140, digital information for receiving input, character information or connect Touch touch operation/non-contact gesture, and generate letter related with the user setting of electronic equipment 100 and function control Number input etc..

Show equipment 110, including display panel 111, for showing information input by user or being supplied to user's Information and the various menu interfaces of electronic equipment 100 etc. are mainly used for showing in electronic equipment 100 in the embodiment of the present application Image to be detected that camera or sensor obtain.Optionally, display panel can use liquid crystal display (liquid Crystal display, LCD) or the forms such as Organic Light Emitting Diode (organic light-emitting diode, OLED) To configure display panel 111.

Electronic equipment 100 can also include one or more sensors 170, such as imaging sensor, infrared sensor, swash Optical sensor, pressure sensor, gyro sensor, baroceptor, Magnetic Sensor, acceleration transducer, range sensor, Close to optical sensor, ambient light sensor, fingerprint sensor, touch sensor, temperature sensor, bone conduction sensor etc., Middle imaging sensor can be flight time (time of flight, TOF) sensor, structured light sensor etc..

In addition to this, electronic equipment 100 can also include for the power supply 150 to other module for power supply.Electronic equipment 100 It can also include less radio-frequency (radio frequency, RF) circuit 180, for carrying out network communication with Wireless Communication Equipment, It can also include WiFi module 190, for carrying out WiFi communication with other equipment, for example, for obtaining other equipment transmission Image or data etc..

Although and it is not shown in FIG. 1, electronic equipment 100 can also include flash lamp, bluetooth module, external interface, press Other possible functional modules such as key, motor, details are not described herein.

Based on above-mentioned introduction, the application provides the method and calculating equipment of a kind of plane monitoring-network, and wherein method can make It calculates equipment and is able to detect that the more than one plane in image data.In the embodiment of the present application, method and calculating equipment are bases The reality of apparatus and method for is calculated since the principle that method and calculating equipment solve the problems, such as is similar in same inventive concept Applying can be with cross-reference, and overlaps will not be repeated.

In the embodiment of the present application, it is described for calculating equipment and being electronic equipment 100, but is not intended to limit this hair Bright embodiment is applied in other kinds of calculating equipment.As shown in fig.2, the detailed process of the plane monitoring-network method can wrap It includes:

Step 201: electronic equipment 100 obtains image data to be processed.

Herein, image data to be processed can be two dimensional image, than as the following examples one in depth image, embodiment The first RGB image and the second RGB image in three, can also be with three dimensional point cloud, the point cloud in two as the following examples, herein It is not specifically limited.

It should be understood that image data to be processed can be electronic equipment shooting, it is also possible to from electronic equipment for depositing It is obtained in the picture library of storage image, is also possible to what other equipment were sent.

Step 202: electronic equipment 100 is split image data to be processed, obtains N number of sub-image data, N be greater than 1 integer.

Step 203: electronic equipment 100 determines corresponding cloud of at least one sub-image data in N number of sub-image data Information.

Step 204: electronic equipment 100 is according to corresponding cloud of at least one sub-image data in N number of sub-image data Information carries out clustering processing to corresponding cloud of N number of sub-image data, obtains K coarse extraction plane；K is just whole no more than N Number.

Step 205: electronic equipment 100 optimizes processing to K coarse extraction plane, obtains plane after L optimization；L is Positive integer no more than K.

In the embodiment of the present application, electronic equipment is split image data to be processed, and to N number of subgraph that segmentation obtains As corresponding cloud progress clustering processing of data, the more than one plane in image data to be processed can detecte.

It describes in detail combined with specific embodiments below to above-mentioned plane monitoring-network method shown in Fig. 2.

Embodiment one

In the embodiment of the present application, the image data to be processed in above-mentioned steps 201 is depth image, and depth image can be with Regard a secondary gray level image as, wherein the gray value of each pixel in gray level image can characterize certain point in scene and exist Projection on the optical axis of camera.It is a kind of schematic diagram of the RGB image of scene provided by the embodiments of the present application referring to Fig. 3 a. In the extraction for carrying out depth information for the scene in the corresponding three-dimensional space of RGB image shown in Fig. 3 a, obtain such as Fig. 3 b institute The depth image shown.

In the embodiment of the present application, depth information can be obtained using modes such as ToF camera, structure light, laser scannings, from And obtain depth image.It should be understood that in the embodiment of the present application can also using it is other it is any can obtain depth image by the way of (or camera) realizes acquisition depth image.Hereinafter only it is illustrated for obtaining depth image using ToF camera, but this The limitation to depth image mode is obtained is not caused, it is subsequent not repeat.

It should be noted that, although point cloud is three-dimensional concept, the pixel in depth image is two-dimensional concept, but known In the case where the depth value that some in two dimensional image is put, the image coordinate of the point can be converted into the seat of the world in three-dimensional space Mark, so, the point cloud in three-dimensional space can be recovered according to depth image.For example, can be completed using multiple view geometry algorithm Image coordinate is converted into world coordinates, specific conversion regime and process are not construed as limiting.

Electronic equipment can carry out plane monitoring-network processing to depth image, specifically after obtaining depth image using ToF It may include following several possible processes.

Process one, is split depth image, and multiple nonoverlapping regions is divided into (each region to be known as son below Image, alternatively referred to as sub- depth image).

If regarding plane monitoring-network method provided in an embodiment of the present invention as a plane detection algorithm process, process One is considered as the initialization procedure of this plane detection algorithm, and what is inputted in this plane detection algorithm is a frame depth map Picture has image coordinate (u, v) and depth value (gray scale) on depth image.This process is split for depth image, is obtained Then subgraph after multiple segmentations can carry out subsequent processing to each subgraph respectively.Due to can be with according to depth information Correspondence recovers a cloud, then carrying out processing correspondence for each subgraph obtained after segmentation can reflect in three dimensions Corresponding cloud of each subgraph is handled, that is to say, that can be used inside algorithm cloud world coordinates (x, y, Z) it is calculated, for example is fitted the calculating of plane, on the one hand can detect multiple planes, another party as far as possible Face may also speed up the processing speed to depth image.

For the depth image shown in Fig. 3 b, the above process one is described in detail below with reference to specific example.

It is split for depth image shown in Fig. 3 b, the signal of the subgraph after obtaining segmentation as shown in Figure 3c Scheme, it is merely exemplary in Fig. 3 c to show the not overlapping region (hereinafter referred to subgraph) of 42 regular sizes after segmentation, it can Choosing, 42 regions after segmentation are also possible to different size of region.It should be understood that can basis during concrete implementation Actual needs is split, and the quantity of specific subgraph is not specifically limited.

In one possible implementation, multiple nonoverlapping regions are split into depth image, it can be by depth The region in image for a regular size is spent, each region may include multiple points, and each point can be mapped to three-dimensional space Between in a number of point composition point cloud, the first moment of the corresponding cloud in these regions can be calculated during initialization And second moment, wherein first moment is the center of gravity of the corresponding cloud in this region, and second moment is the side of the corresponding cloud in this region Difference.

After obtaining 42 subgraphs as shown in Figure 3c, the subsequent processing for 42 subgraphs is (such as at cluster Reason) it can be there are many implementation.In a kind of possible implementation, for corresponding cloud of each subgraph, carry out respectively Such as the clustering processing in following processes two.It is needed in this mode at all subgraphs obtained after Range Image Segmentation Reason, compared to the prior art in from center begin stepping through image to be processed until detecting that a plane just stops detecting, can only Detect that the scheme of a plane, scheme provided by the present application are split to obtain subgraph to entire image to be processed, and right All subgraphs carry out clustering processing, so as to detect multiple planes.

It, can be first from 42 sons in alternatively possible implementation in order to further speed up the speed of plane monitoring-network It determines that corresponding cloud may be fitted to the subgraph of a plane in image, is then fitted to one for the possibility determined The subgraph of a plane carries out the clustering processing such as following processes two, so there is no need to for a possibility that being fitted to plane compared with Low subgraph carries out subsequent processing, so as to save the time of entire detection planarization process.

In one example, for subgraph shown in Fig. 3 c, graph structure can be constructed, graph structure is a kind of computer number According to structure, using the information of corresponding cloud of a subgraph as a node of graph structure, for example, corresponding with subgraph B3 For the information of point cloud is as a node in graph structure, subgraph B3 corresponding node in graph structure includes the subgraph As the first moment, second moment (also referred to as mean square error) and the calculated normal vector of plane fitting of corresponding cloud of B3.Then It will likely be fitted to one side of construction between two nodes of plane, carry out subsequent processing for the node for having side.

In the embodiment of the present application, it is equal with the quadratic sum of the error of initial data corresponding points that mean square error is fitting data Value, that is to say, that for being fitted to obtain a plane A according to all the points for including in cloud, for some point in cloud For, the mean square error for putting cloud is the mean value of the quadratic sum of distance between all the points and plane A in point cloud.About the square of cloud Error is not repeating hereinafter.

In one possible implementation, two nodes of construction can be chosen whether according to the relationship between two nodes Between side.Specifically, corresponding cloud of each node carries out that corresponding normal vector can be obtained after plane fitting, such as Angle is greater than angle threshold value between the normal vector of the corresponding fit Plane of two nodes of fruit, just not structure between the two nodes Side is made, if angle is less than or equal to angle threshold value between the normal vector of two nodes, just constructs side between the two nodes.So Afterwards, the clustering processing that process two is carried out for the node for having side there is not the node on side without cluster.

Referring to Fig. 3 d, schematic diagram is initialized for graph structure provided by the embodiments of the present application.

As shown in Figure 3d, a classification is done for the corresponding all node of graph of depth map, wherein black circles expression does not have The node of depth value, × it is the discontinuous node of depth value, two node edges are discontinuous, and solid black point indicates subgraph pair The smaller node of the mean square error (mean square error, MSE) that the point cloud plane fitting answered comes out.It constructs While be exactly in fact graph structure while, it connects two adjacent nodes, indicates there is relationship between the two nodes, does not connect and is exactly It is not related.Assuming that connected two nodes are A node and B node, then connection side needs to meet following condition: condition 1, The depth value of any of A node and B node corresponding cloud of node is continuous, i.e., A node and B node are all not without depth value Node, nor the discontinuous this node of depth value；Normal vector folder between condition 2, A node and B node the two nodes Angle is less than angle threshold value.

Process two carries out clustering processing for the subgraph in process one.

Only for node (i.e. darker regions in Fig. 3 d) the progress clustering processing for having side shown in Fig. 3 d in process two For be illustrated.

It, can be (hereinafter referred to as minimum by establishing minimum heap data structure in order to further speed up the speed of clustering processing Heap) to realize, preferentially the node progress clustering processing to most possible one plane of formation, most rickle can be regarded as section The process that the processing sequence of point is ranked up, can prioritize processing the smallest node of second moment by most rickle, most from second moment Small node is begun stepping through.

In one example, a most rickle is initially set up, all nodes all respectively become one kind in figure at this time.To each A node carries out plane fitting, and each node fitting result corresponds to a fitting mean square error, and big according to fitting mean square error Each node is pressed into most rickle by small sequence from big to small, and the heap top (object popped up every time) of most rickle is lowest mean square The corresponding node of error.It being iterated later, iteration pop-up each time is all the current node with minimum error of fitting, this A node is merged with some node of surrounding, the fused mean square error of available two nodes, after selection fusion Two nodes before the square the smallest fused node replacement of error, are added graph structure, and rejoin most rickle.After fusion The mean square error of node if it is less than or be equal to first threshold, then by the fusion posterior nodal point be added most rickle in, if the fusion The mean square error of posterior nodal point is greater than first threshold, then abandons the fusion posterior nodal point.The stop condition of iteration is all sections in figure The plane fitting mean square error of point between any two is greater than first threshold.

Process two is described in detail below.

S1 carries out plane fitting according to corresponding cloud of each node, for example can use principal component analysis (principal component analysis, PCA) algorithm carries out plane fitting, cardinal principle to cloud are as follows: assuming that section Point all in corresponding cloud is put in one plane, then the characteristic value of Calculation Plane, if having one in three characteristic values A characteristic value is less than threshold value, it may be said that the point for including in bright cloud in one plane, should be corresponding less than the characteristic value of threshold value special Sign vector is exactly the normal vector of plane.Then a mean square error can be calculated according to all points for being fitted to plane, this is Square error is the mean value of the quadratic sum of distance between all the points and the plane of fitting in corresponding cloud of the node.One node pair A mean square error is answered, and node is pressed into according to the sequence from big to small of mean square error by most rickle according to calculated result.

S2 takes out the corresponding node of least mean-square error from most rickle, by the corresponding node of the least mean-square error with The adjacent node of the node is merged, and the mean square error of fusion posterior nodal point is calculated.

For 9 nodes marked below with reference to dotted line frame shown in Fig. 3 c, node fusion process illustrate It is bright.

As shown in 3b, 9 nodes that dotted line frame is marked are respectively node B3, node B4, node B5, node C3, node C4, node C5, node D3, node D4, node D5, it is assumed that this 9 nodes are all the nodes for having side, corresponding to this 9 nodes Point cloud carries out plane fitting respectively and obtains mean square error, it is assumed that the corresponding mean square error of node C4 is minimum, has 8 around node C4 A node B3, node B4, node B5, node C3, node C5, node D3, node D4, node D5, by node C4 respectively with surrounding 8 nodes in each node merged, available fused 8 new nodes, respectively node C4B3, node C4B4, node C4B5, node C4C3, node C4C5, node C4D3, node C4D4, node C4D5, it is fused for this 8 Corresponding cloud of the fused new node of each of new node carries out plane fitting and obtains mean square error, it is assumed that fused new The corresponding mean square error of node interior joint C4B4 is minimum in the corresponding mean square error of above-mentioned 8 fused new nodes and should The corresponding mean square error of node C4B4 is less than or equal to first threshold, then selection node C4B4 replaces node C4 and node B4 to add Enter graph structure, and node C4B4 is added to most rickle.That is, dotted line frame is marked in Fig. 3 c after fusion process 9 nodes of note, become such as 8 nodes that dotted line frame is marked in Fig. 3 e, respectively node B3, node B5, node C3, node C4B4, node C5, node D3, node D4, node D5.It then proceedes to 8 nodes in Fig. 3 e according to such as above-mentioned for node The fusion process of C4 and surroundings nodes is clustered, until meeting the stop condition of iteration.The stop condition of iteration is institute The mean square error that the corresponding cloud of new node for having in node the fusion between node two-by-two obtain is obtained by plane fitting is all Greater than first threshold.

Herein, there can be at least two possible modes to determine whether aggregators C4 and node B4.

In one of possible mode, egress C4 8 new sections that 8 nodes merge with around respectively are being determined After the corresponding mean square error of point, first obtain 8 mean square errors are made comparisons with first threshold respectively, if this 8 new sections There are the nodes that mean square error is less than or equal to first threshold in point, then being less than or equal to the section of first threshold from mean square error Determine that the corresponding node of least mean-square error is added to most rickle in point.If mean square error is not present in this 8 new nodes Less than or equal to the new node of first threshold, then illustrating that node C4 cannot be merged with surroundings nodes, that is to say, that pass through this After fusion process, 9 nodes that as a result still dotted line frame as shown in Figure 3c is marked.

In alternatively possible mode, egress C4 8 new sections that 8 nodes merge with around respectively are being determined After the corresponding mean square error of point, it is ranked up, is determined in this 8 new nodes against the corresponding mean square error of 8 new nodes The corresponding new node of least mean-square error makes comparisons the least mean-square error with first threshold, if the least mean-square error Less than or equal to first threshold, then being added the corresponding new node of the least mean-square error to most rickle.If the minimum is equal Square error is greater than first threshold, then illustrating that node C4 cannot be merged with surroundings nodes, that is to say, that passes through this fusion process Later, 9 nodes that as a result still dotted line frame as shown in Figure 3c is marked.

It should be noted that by being illustrated for handling and having the node on side in this present embodiment, so be directed to one It is a have while node C4 around if it is none of it is other have while node, there is no need to carry out to this node C4 for that Fusion process is stated, if there are other nodes for having side around node C4, can centainly be found around node C4 fused New node meets the node that mean square error is less than or equal to first threshold.Certainly, the embodiment of the present application can also be for such as Fig. 3 c Shown in divide after subgraph directly carry out clustering processing, then being possible to node C4 occur cannot to save with around any one The case where point is merged.

In above-mentioned cluster process, above-mentioned clustering processing successively is carried out from be hit by a bullet out the node of least mean-square error of most rickle, If the node of the least mean-square error has the node that can be merged, just fused node is added into most rickle, If the node of the least mean-square error is added the node that the node of the least mean-square error can not merge again Into most rickle, until each node in some moment most rickle can not find the node that can be merged, with regard to stopping clustering Journey.At this point, corresponding cloud of each node in most rickle is on different surfaces, i.e., any two node can not be fused into one Plane, therefore, each node in most rickle in remaining each node can correspond to a plane.

It in above-described embodiment, is illustrated in the clustering processing mode of single thread as an example, that is to say, that be directed to depth Corresponding 42 nodes of image are added to most rickle, pop up a node every time, find from the node around the node With the node that can merge the node, after obtaining fusion posterior nodal point, fusion posterior nodal point is put into most rickle, is then proceeded to Foregoing fusion process is carried out from the most rickle the smallest node of mean square error of being hit by a bullet out, until cluster process terminates.In this mode In, it is begun stepping through from a node, finally obtains multiple planes.

It should be understood that the embodiment of the present application can also detect multiple planes using the clustering processing mode of multithreading, such as with Using the clustering processing mode of three threads, at this moment once pop up 3 the smallest nodes of mean square error from most rickle, pop-up it is every A node corresponds to a thread, and the fusion process of per thread can refer to the fusion process of above-mentioned three thread in these three threads. It is all added into most rickle in the fusion posterior nodal point of three threads, then proceedes to 3 mean square error minimums of being hit by a bullet out from most rickle Node, foregoing fusion process is carried out with surroundings nodes, until cluster process terminates.This mode may be implemented simultaneously from multiple Node is begun stepping through, and finally obtains multiple planes, compared to the above-mentioned mode from a node traverses, the processing speed of this mode It spends faster, so as to faster obtain the plane monitoring-network result of entire image to be processed.

For have side shown in Fig. 3 d node carry out process two in clustering processing after, it is available as illustrated in figure 3f Cluster result, i.e., if the figure of grey in Fig. 3 f to join together is coarse extraction plane.

It there may be two coarse extraction planes is actually conplane in multiple coarse extraction planes that process two obtains Situation, it is also possible to there is the point for being not belonging to the coarse extraction plane in coarse extraction plane, it is also possible to have other around coarse extraction plane Point belong to the coarse extraction plane, that is to say, that the coarse extraction plane may miss some points for belonging to the coarse extraction plane. More accurate plane monitoring-network is as a result, optimize processing to coarse extraction plane in order to obtain, specifically may refer to following processes three, Process four and process five.

For the plane further refined, avoid in coarse extraction plane exist be not belonging to the coarse extraction plane Point can carry out erosion algorithm operation for the coarse extraction plane, and specific implementation is referring to process three.

Process three, the coarse extraction plane obtained to clustering processing in the above process two carry out marginal operation, can also claim For erosion algorithm operation.

The edge of coarse extraction plane obtained in process two is zigzag fashion, and according to the precision of graph structure, this is thick The edge for extracting plane will cover different regions, and may intervene the region of another plane.Etching operation, which can reduce, works as The domination region of each node in preceding figure, and leave the region in each plane bosom.

Based on Fig. 3 f, coarse extraction plane therein is marked, obtain such as Fig. 3 g thus to mark each coarse extraction flat The image in face, it should be appreciated that coarse extraction plane be marked this movement be not detect planarization process in one it is necessary Movement is merely for convenience of illustrating the coarse extraction plane obtained by process two.

As Fig. 3 g illustrates the coarse extraction plane A that dotted line frame marked, coarse extraction plane B, coarse extraction plane C, thick Extract plane D, coarse extraction plane E, coarse extraction plane F.

By taking the coarse extraction plane A that dotted line frame in Fig. 3 g is marked as an example, etching operation is carried out to coarse extraction plane A and is removed slightly Coarse extraction plane A can be avoided so that obtained plane is more accurate as far as possible by extracting the node that plane A jagged edges one enclose In include being not belonging to the point of the coarse extraction plane.

Coarse extraction Planar Contraction can be made to arrive for the etching operation of the coarse extraction plane that process two obtains, the above process three The circle of edge one is removed in center, for example the coarse extraction plane A in comparative diagram 3a and Fig. 3 g, Fig. 3 g is the desktop in Fig. 3 a, In The depth value tomography at table angle in Fig. 3 g, if the erosion algorithm without process three operates, that is to say, that in the above process The region growings operation of process four is directly carried out after two, it is easy to cause coarse extraction plane A and following coarse extraction plane C, Coarse extraction plane D etc. carries out region growing together, but actually coarse extraction plane A and following coarse extraction plane C, coarse extraction Plane D is not approximately the same plane, if together carry out region growing will obtain error result, so to coarse extraction plane into It, can be to avoid such case after row etching operation.

For the plane further refined, avoiding each coarse extraction plane that from may missing, some to belong to this thick The point for extracting plane can carry out algorithm of region growing operation for the coarse extraction plane A, specifically by taking coarse extraction plane A as an example Implementation is referring to process four.

Process four is further operated using algorithm of region growing for the coarse extraction plane by etching operation, to avoid Omit the point for belonging to the coarse extraction plane.

It, can be simultaneously using multiple coarse extraction planes as center region, to outgrowth in a kind of possible implementation.

Below with reference to specific example, to how realizing that region growing is described in detail.

To be for growth outward is realized in center region, due to being grasped through excessive erosion by the coarse extraction plane A of etching operation The coarse extraction plane A of work is removed obtained from marginal operation for coarse extraction plane A, so by the thick of etching operation Extracting plane A is not coarse extraction plane A, below for the coarse extraction plane A of etching operation is known as coarse extraction plane A' It is illustrated.

Using coarse extraction plane A' as center region to outgrowth during, need to every around coarse extraction plane A' A pixel (these pixels are not in coarse extraction plane A') carries out judging whether to be divided to coarse extraction plane A', is Convenient for explanation, it will judge whether that this pixel that can be divided to coarse extraction plane A' is known as operating point.

In the embodiment of the present application, there are many may determine that whether operating point can be divided to the realization of coarse extraction plane A' Mode.

In a kind of possible implementation, if there are 50% or more neighbours in all vicinity points around operating point Nearly pixel belongs to some coarse extraction plane, such as coarse extraction plane A', puts down then the operating point is also divided to the coarse extraction Face A'.

By taking operating point is pixel K as an example, if there is 8 vicinity points around pixel K, if this 8 neighbouring pictures There are more than 4 pixels in vegetarian refreshments and belong to coarse extraction plane A', for example there are 5 vicinity points to belong to slightly around pixel K Plane A' is extracted, then the corresponding point cloud data of pixel K is just substituting to the corresponding plane fitting side coarse extraction plane A' Cheng Zhong, to verify whether pixel K belongs to coarse extraction plane A', if verification result is to belong to the coarse extraction plane A', then pixel K is just divided to coarse extraction plane A'；If verification result is to be not belonging to coarse extraction plane A', It so abandons for pixel K being divided to coarse extraction plane A'.

In alternatively possible implementation, if there are 50% neighbours in all vicinity points around the operating point Nearly pixel belongs to coarse extraction plane A', and 50% vicinity points belong to coarse extraction plane C', at this time can be by by the behaviour Make point and carry out plane fitting with the vicinity points for belonging to two coarse extraction planes respectively, pixel K is divided to quasi- by selection Close coarse extraction plane belonging to the corresponding smaller vicinity points of mean square error of result.

By taking operating point is pixel K as an example, if there is 8 vicinity points around pixel K, if 4 pixel categories In coarse extraction plane A', then pixel K is substituted into coarse extraction plane A''s close to coarse extraction plane C' by 4 pixels respectively The plane fitting equation of plane fitting equation and coarse extraction plane C' compares the substitution of two plane fitting equations as a result, later Pixel K is divided in the better coarse extraction plane of result in the substitution result of two coarse extraction planes (A' and C').

By the above process four, it is around each coarse extraction plane that process three can be obtained and with some coarse extraction The conplane neighbor point that belongs to of plane is divided to the coarse extraction plane, so that the plane that detected is more complete It is whole.

The above process two, process three, multiple planes that any one process detects in process four, are all based on to two Obtained from the understanding for tieing up image, there may be visual error, for example there may be two planes are actually same A plane, but what is reflected on 2d may not be a plane.In order to avoid there is such case, can pass through Following processes five, which are realized, eliminates multiple planes collimation error that may be present that the several processes in front detect.

Process five merges plane operations to multiple planes, is referred to as merging Plat algorithm.

Process five can be adapted for carrying out for the result that any process obtains in the above process two, process three and process four Merge plane operations, is illustrated so that the result obtained for process four merges plane operations as an example below.

It can be by process four using primary bottom-up hierarchical clustering algorithm for multiple planes that process four obtains Obtained in all planes once traversed, merge any two spatially plane similar in space length.

In one possible implementation, plane two-by-two in the multiple planes that can be obtained using PCA algorithmic procedure four Plane fitting is carried out, for example multiple planes that process four obtains are plane A', B', C', D', E', F', two-by-two after plane fitting The plane DE's that plane BC and plane D' that available plane B', plane C' are fitted, plane E' are fitted is square Error is less than threshold value, then plane B' and plane C' are in one plane, plane D' and plane E' in one plane, then may be used Continuing plane BC, plane DE together with remaining plane A', F', continue to be fitted between plane two-by-two, it is known that any The mean square error for the plane being fitted between two planes is both greater than threshold value, and fitting terminates, and all planes remaining at this time are not one In a plane.

In one implementation, can first judge whether any two plane is parallel, then judge that parallel two are flat Whether the space length of face spatially is close.

In some embodiments, normal vector that two planes can be calculated, by determining whether normal vector determines in parallel Whether two planes are parallel, if the normal vector of two planes is parallel, two planes are also parallel, illustratively, A pairs of plane The normal vector answered is n1=(A1, B1, C1), and the corresponding normal vector of plane A is n2=(A2, B2, C2), if A1/A2=B1/ B2 and B1/B2=C1/C2 and A1/A2, B1/B2, C1/C2 are same constant, then plane A is parallel with plane B, herein, "/" is indicated divided by A1/A2 indicates A1 divided by A2.

In an implementation, if it is determined that it is not parallel to go out two planes, then the two planes will not be judged as same put down Face, the two planes will not be judged as the similar plane on space length in other words；If it is determined that two planes are parallel, So the direction of the common vertical line of the two planes can be determined by the way that there are vertical ranges between the two determining planes (the namely direction of normal vector) illustrates variance with regard to big if the distance on common vertical line direction of 2 planes is very big；If Two planes on common vertical line direction apart from very little, illustrate variance with regard to small, it is possible to by PCA algorithm to two planes into Row fitting, calculates mean square error, then decentralization obtains variance, then determines two planes by determining variance size Whether distance is close, for example variance is less than or equal to variance threshold values, illustrates the closely located of two planes, the two are flat in other words Face is in one plane.

In one example, the merging plane operations in such as process five are carried out based on each plane in Fig. 3 g, can obtained Plane after to merging as illustrated in figure 3h.Wherein by Fig. 3 g coarse extraction plane D and coarse extraction plane E merge into it is same flat Face, coarse extraction plane B and coarse extraction plane C merge into approximately the same plane, and wherein coarse extraction plane A is a plane, coarse extraction Plane F is a plane.

By above-mentioned five processes, may be implemented quickly to being traversed in entire depth image, to quickly detect To the plane of multiple precisions.

It should be noted that above-mentioned five processes can be used alone, also can choose one or more processes and combine makes With.For example, process three is omitted only with Four processes such as process one, process two, process four, processes five.It can certainly be by Other methods for combining to realize above-mentioned plane monitoring-network.Illustratively, the above process three can also process four or process five it After carry out.

In the examples below, plane monitoring-network method may include multiple processes, and between any two processes there may be The case where nesting executes illustratively can nested implementation procedure A in process one.

Embodiment two

In the embodiment of the present application, the image data to be processed in above-mentioned steps 201 can be three dimensional point cloud.This Shen Please embodiment can be adapted for orderly putting cloud, be readily applicable to unordered cloud.Orderly all the points in point cloud are arranged in order, It can be easily found the consecutive points information of each point, the point arrangement in unordered cloud is unordered, general nothing without rule Method finds the consecutive points information of each point.

For the scene shown in Fig. 3 a, by establishing Octree knot to point cloud corresponding in scene as shown in Figure 3a Structure carries out plane fitting to the point cloud in 8 nodes in one layer in conjunction with plane fitting mode, then according to each node respectively In all the points whether in approximately the same plane determine whether to continue to divide the point cloud in the node layer, final detection obtains multiple Refine plane.

In the extraction for carrying out depth information for scene shown in Fig. 3 a, can be swept using ToF camera, structure light, laser The modes such as retouch to obtain depth information, to obtain depth image, the related content of depth image be may refer in embodiment one Associated description, details are not described herein again.

Process one, the octree index of building point cloud, is split three-dimensional point cloud with realizing.

In one possible implementation, the point cloud in a scene is subjected to decentralization, and according to different points Resolution is divided by slightly to the set of essence.The plane for the point cloud that resolution ratio herein includes by the corresponding voxel of octree nodes is intended It closes result to determine, wherein each octree nodes characterizes the set at a voxel midpoint.During constructing Octree, The first order and second order moments that this node corresponds to voxel midpoint cloud are calculated while constructing each node.

Below with reference to Fig. 4 a and Fig. 4 b, describe in detail to how to construct octree structure.

It is illustrated first against octree structure, octree structure is by a kind of data model, and octree structure passes through Volume elements subdivision is carried out to the geometry entity of three-dimensional space, each volume elements Time & Space Complexity having the same passes through circulation Recursive division methods carry out subdivision to the geometric object for the three-dimensional space that size is (2n*2n*2n) (2n*2n*2n), thus structure At a directional diagram with root node.If divided volume elements attribute having the same in octree structure, such as phase Same attribute can be for the point in volume elements in the same plane, then the volume elements constitutes a leaf node, otherwise continues to the body Member is split into 8 sub-cubes, successively passs subdivision, for the spatial object of (2n*2n*2n) (2n*2n*2n) size, at most cuts open Divide nn times.

A referring to fig. 4 is the subdivision schematic diagram provided by the embodiments of the present application to three-dimensional space.As shown in fig. 4 a, to three-dimensional The geometric object in space carries out subdivision, and subdivision result can reflect in the octree structure shown in Fig. 4 b.For example, in Fig. 4 a In, eight an equal amount of sub-cube B1, B2, B3 ... B8 are divided into a cube A in Fig. 4 a, reflection is in fig. 4b For eight child node b1, b2, b3 ... b8 of a root node a connection.Continue to be split into eight for sub-cube B3 in Fig. 4 a An equal amount of sub-cube C1, C2, C3 ... C8, reflection in fig. 4b for eight child node c1, c2 of child node b3 connection, c3……c8.Continue to be split into eight an equal amount of sub-cube D1, D2, D3 ... D8 for sub-cube B8 in Fig. 4 a, Reflection is eight child node d1, d2, d3 ... d8 of child node b3 connection in fig. 4b.

Each child node can characterize the set at a voxel midpoint in above-mentioned Octree, may include one in a voxel A or multiple points.

It is by carrying out corresponding cloud of depth image by slightly to essence for the octree structure that the above process one constructs Segmentation obtains the Octree child node of each level from top to bottom, and as shown in Figure 4 b, the first level includes child node a, the second layer Grade includes child node b1, b2, b3 ... b8.Third level includes child node c1, c2, c3 ... c8, d1, d2, d3 ... d8.Needle To each child node in each layer, if whether the point for continuing to divide be by including in corresponding cloud of the child node is same In a plane, such as in Fig. 4 b, corresponding cloud of child node b1 in the same plane, so do not need to child node b1 after Continuous subdivision, and the point that the corresponding voxel of child node b3 includes is not in the same plane, so needing to continue to child node b3 It carries out continuing to divide, obtains child node c1, c2, c3 ... c8 in next level.To the last every height in a level The point for including in corresponding cloud of node is all in one plane, alternatively, the quantity in point cloud including point is less than amount threshold, Do not continuing to divide the child node in the level.

Whether need to continue subdivision for each sub-cube in each level, can determine whether the node pair in each level The point cloud answered whether in one plane, if the point for including in sub-cube not in one plane, needs to continue to divide The sub-cube；If the point for including in sub-cube is in one plane, not continuing to divide the sub-cube can be with It is realized by following procedure A.It should be noted that when the quantity for the point for including in sub-cube is less than amount threshold, i.e., The point for including in sub-cube is set in one plane, also not continue to divide the sub-cube.

Process A, the normal vector of each voxel of rapidly extracting judge whether voxel needs to continue to divide.

The normal vector that the corresponding voxel of each child node is calculated since the thicker level in octree structure, is then pressed According to by slightly to the normal direction for the corresponding voxel of eight octree nodes of the sequence level-one level-one of essence calculated downwards in each level Amount.

In a kind of example, for the corresponding voxel of each octree nodes, voxel can be calculated using PCA algorithm Normal vector, such as the corresponding voxel of node b1, node b1 correspond to a voxel, the three-dimensional point cloud for including in the voxel corresponding 3 A characteristic value, for example be r1, r2, r3, the corresponding feature vector of each characteristic value, that is to say, that include three in the voxel Corresponding 3 feature vectors of dimension point cloud, then determine to be worth the smallest characteristic value from three characteristic values r1, r2, r3, for example r2 is most It is small, then the corresponding feature vector of characteristic value r2 is the normal vector for the three-dimensional point cloud for including in the voxel.

After the normal vector (the as corresponding feature vector of minimal eigenvalue) for calculating the corresponding voxel of node b1, The quotient between the sum of this feature value r2 and three characteristic values (i.e. the sum of r1, r2, r3 three) is calculated, by the quotient and third Threshold value is compared, if the quotient is less than or equal to the third threshold value, include in the corresponding voxel of node b1 three Point cloud is tieed up in the same plane, it is also assumed that the three-dimensional point cloud for including in the corresponding voxel of node b1 is a plane. If the quotient is greater than the third threshold value, the three-dimensional point cloud for including in the corresponding voxel of node b1 is not in approximately the same plane On, it is also assumed that the three-dimensional point cloud for including in the corresponding voxel of node b1 is not a plane, it may be possible to constitute multiple Plane needs node b1 to continue eight child nodes for being divided into next stage in this case, eight child nodes of next stage after It is continuous to determine the need for continuing to segment in the manner described above.For each child node, if meeting the condition of convergence, so that it may no longer Subdivision, the condition of convergence can be any one of the following conditions: condition one, and the quantity for the point for including in the corresponding voxel of node is small In amount threshold；Condition two, the quotient between the sum of the corresponding characteristic value of normal vector of the corresponding voxel of node and other feature value Value is greater than third threshold value.

Since minimal eigenvalue (namely mean square error) can actually reflect the degree of fluctuation put on normal vector direction, If this minimal eigenvalue is less than or equal to second threshold, illustrate that the degree of fluctuation put on normal vector direction is small, also Illustrate that node b1 is a plane.

A by the above process can count eight child nodes in a level simultaneously for a level Algorithm vector, determines whether the point cloud for including in the corresponding voxel of each byte point needs to continue to divide, so as to quickly complete At the cutting procedure of eight child nodes of a level, for a child node, pass through above-mentioned rapidly extracting child node pair The normal vector answered may be implemented quickly to determine whether the point cloud for including in the corresponding voxel of a node is a plane, from And it can quickly determine the need for continuing to segment the point cloud for including in the corresponding voxel of the node.

Octree structure is being obtained by process A, the point that the corresponding voxel of the octree nodes in each level includes In one plane, that is to say, that the corresponding normal vector of the corresponding voxel of each octree nodes, it later can be by from each The voxel with coplanar relation is determined in the corresponding voxel of a node, the voxel with coplanar relation is merged, and is determined The concrete mode of coplanar relation may refer to following processes three.

Process two corresponds between each voxel whether have coplanar relation for node layer each in octree structure.

In a kind of possible implementation, using the corresponding body of node each in each layer of Hough (Hough) transformation calculations Whether element has coplanar relation, and the duality relation of the sample space and parameter space that specifically can use Hough transform is realized.

Wherein, sample space is the point cloud of the corresponding voxel packet of each octree nodes as obtained in aforementioned process A What normal vector was constituted.One normal vector corresponding one is crossed the plane of origin, then the normal vector then corresponds to one in parameter space A point.Illustratively, the corresponding plane equation of a voxel in sample space is ax+by+cz=1, is considered as sampled point It is the relationship of antithesis between (x, y, z) and normal vector (a, b, c).

If the normal vector of the corresponding plane equation of a voxel is converted by Hough (Hough) transformation in parameter space In a point, then the point that the normal vector for belonging to conplane voxel is formed in parameter space may build up together, These points together will tend to a central point in parameter space, that is to say, that other are in this central point Surrounding fluctuates.With the increase of sampled point, the normal vector of many voxels is also just acquired, the result of parameter space will have centainly Statistical property, that is to say, that with the increase of sampled point, these sampled points, which can be presented, levels off to the characteristic at some center, i.e., Closer to the center, sampled point is more, statistical property possessed by the result here it is parameter space.Due to coplanar relation The corresponding normal vector of each voxel be parallel to each other, so the statistical property being had according to the result of parameter space can obtain The set of coplanar voxel object.

In some embodiments, for the normal direction of the corresponding voxel of the child node for belonging to same level in octree structure Amount can determine in the result of parameter space there is the coplanar corresponding set of voxel, specific mistake by K-means algorithm Journey is as follows: the aggregation central point in parameter space is got first, than if any 3 aggregation central points, then in each aggregation Centered on heart point, a ball is drawn by the radius of a ball of preset value, the point in ball is the set with the voxel of coplanar relation, The specific value of the preset value is not construed as limiting, and can be configured according to actual needs.

It obtains in octree structure in the voxel of same level child node, has for by Hough (Hough) mapping mode There is the set of the voxel of coplanar relation, so that it may obtain the corresponding plane equation of set of these voxels with coplanar relation.

By the available multiple coarse extraction planes of above-mentioned Hough (Hough) conversion process, each coarse extraction plane is corresponding One plane equation.

It, can be by clustering processing mode to level of child nodes pair each in octree structure in alternatively possible implementation The voxel answered is clustered, child node c1, c2, c3 ... the c8 and d1, d2, d3 ... for including with third level in Fig. 4 b For d8, the corresponding voxel of 16 child nodes for including in the third level is clustered, specific cluster process can refer to above-mentioned Clustering processing mode in embodiment one.

It, can be with for being clustered below with reference to example to 16 child nodes that the third level in octree structure includes From by taking a child node c1 as an example, this child node c1 is melted with other 15 child nodes respectively in 16 child nodes It closes, finds out a child node from other 15 child nodes in addition to child node c1 and melt with what child node c1 was merged The corresponding mean square error of child node is minimum after conjunction, for example child node is corresponding after the fusion merged with child node c1 found out The smallest child node of mean square error be c3, and the smallest mean square error is less than or equal to first threshold, then can will be sub The point-cloud fitting that the point cloud and the corresponding voxel of child node c3 that the corresponding voxel of node c1 includes include is at a plane.This melts Conjunction process iteration is multiple, until can not find the child node that can be merged two-by-two.Fusion results are obtained obtaining third level Later, fusion results are merged with child node each in the second level (child node b1, b2, b4 ... b7).Again will later The fusion results of each child node of second level are merged with the node of the first level, it is last it is available between any two not The node that can be merged, by above-mentioned cluster process, available multiple coarse extraction planes.

The coarse extraction plane obtained in process two is the fitting result of a maximum probability, is inaccurate, if it is eight The fitting result that calculates in the high-level volume elements of tree construction is pitched, then this is the result is that very inaccurate.For example, process two obtains Coarse extraction plane may have an outlier, that is, partial dot is in this coarse extraction plane, but not in Hough transform It in obtained coarse extraction plane, needs to reject these outliers at this time, for the plane further refined, can be directed to The coarse extraction plane carries out RANSAC operation, and specific implementation is referring to process four.

Process three, based on random sampling consistency (random sample consensus, RANSAC) algorithm to process two The obtained corresponding plane equation of coarse extraction plane is refined.

In one possible implementation, it is based on the thick fitting result of Hough transform (i.e. process two), is become with Hough For the coarse extraction plane B got in return, randomly selects the point for including in one group of point and coarse extraction plane B and be fitted one again Plane, for example be plane X1, wherein be the point in plane that Hough transform mode is fitted, if plane X1 is corresponding square Error is greater than the corresponding mean square error of coarse extraction plane B, then this group point is not the interior point of coarse extraction plane B, then again One group of point and coarse extraction plane B is selected to continue to be fitted a plane；If the corresponding mean square error of plane X1 is less than or equal to The corresponding mean square error of coarse extraction plane B, then intra-office (inlier) point that this group point is coarse extraction plane B, later, with this Subject to plane X1, continue that another group of point is added into plane X1, then is fitted a plane, such as plane X2 again；If plane The corresponding mean square error of X2 is less than or equal to the corresponding mean square error of plane X1, then being subject to plane X2, continues to plane Other points are added in X2, then are fitted a plane again, the plane after optimization is finally obtained with this iterative processing.

In alternatively possible implementation, it can be based on the thick fitting result of Hough transform (i.e. process two), with thick For extracting plane C, corresponding equation is plane equation, takes outside coarse extraction plane C and substitutes into plane equation progress Verify whether the point is intra-office point, wherein intra-office point refers to the point shoulding be in coarse extraction plane C；If the point intra-office point, Then the point is added into coarse extraction plane C, if the point is not intra-office point, abandons the point, is continued flat in the coarse extraction Whether be intra-office point, such iteration is excellent to finally obtaining if other points being taken to be substituting to the plane equation to verify other points outside the C of face Plane after change.

In multiple planes that process three obtains, may have two planes is a plane in fact, it is possible to mistake Multiple planes that journey three obtains carry out implementing process referring to the phase of process five in above-described embodiment one as merged plane operations Hold inside the Pass.

Embodiment three

In the embodiment of the present application, the image data to be processed in above-mentioned steps 201 may be RGB image.

In one example, can shoot Same Scene or so two images, this two images by binocular camera is RGB image, wherein can be according to left figure (hereinafter referred to as A figure) and right figure (hereinafter referred to as B figure), it can be with vertical for A figure and B figure Body matching algorithm (such as binocular solid matching (Stereo Matching)) obtains disparity map, and then available depth image. Then plane fitting can be carried out to cloud according to depth image, to realize the purpose of detection plane.

Multiple planes are quickly detected in order to realize, to the two width figure of left and right of binocular camera shooting in the embodiment of the present application As being split processing respectively, segmentation result is obtained.For example the corresponding face block of target object is obtained using semantic segmentation mode, then For example, handling to obtain multiple face blocks with similar quality using image pre-segmentation.

For carrying out semantic segmentation processing for A figure, the segmentation result that progress semantic segmentation obtains is schemed for A are as follows: tool There is image after the segmentation of at least one target object, for example the semanteme being arranged is people, then can be obtained by the mark for being labelled with people The people that note frame is outlined, the corresponding region of this callout box includes target object: people, then by this corresponding region of annotation frame Then referred to as subgraph carries out subsequent processing at least one subgraph that semantic segmentation is handled.For example, by A figure point The multiple subgraphs obtained after cutting, then handle to obtain parallax with the multiple subgraphs obtained after B figure segmentation are carried out Stereo matching Figure, then determines depth map further according to disparity map.

For carrying out image pre-segmentation processing for A figure, the segmentation result that is split for A figure are as follows: including It is multiple to be alternatively referred to as subgraph for image after the segmentation in the region (also referred to as face block) of plane, each face block, then be directed to Each subgraph carries out subsequent processing.For example, will obtained multiple subgraphs after A figure segmentation, such as A1, A2 and A3, then with by B The multiple subgraphs obtained after figure segmentation, such as B1, B2 and B3 carry out Stereo Matching Algorithm and handle to obtain disparity map, then again Depth map is determined according to disparity map.Wherein, neutron image A1 and subgraph B1 is in different points of view in Same Scene Same place band of position C is shot, so can finally obtain the position area according to subgraph A1 and subgraph B1 The corresponding depth map of domain C, the corresponding depth map of this band of position C are known as sub- depth image.

After obtaining multiple sub- depth images, every sub- depth image can be reverted into the point cloud in three-dimensional space, Then the sub- depth image that corresponding cloud can be fitted to a plane can be filtered out from multiple sub- depth images, then Clustering processing is carried out for the sub- depth image filtered out, finally obtains multiple planes.

Below with reference to specific example, the plane monitoring-network method in embodiment three is introduced.

Referring to Fig. 5, for another plane monitoring-network method process schematic provided by the embodiments of the present application.As shown in figure 5, should Plane monitoring-network method includes the following steps:

Step S1 obtains binocular camera and shoots to obtain left (A figure), right (B figure) two width RGB images.

Step S2 carries out semantic segmentation to A figure and B figure respectively or image pre-segmentation is handled, obtains left and right two width RGB The corresponding segmentation result of image.

Wherein, semantic segmentation can be there are many possible implementation, and a kind of possible partitioning scheme is to use full convolution Network implementations coarse segmentation.

It before carrying out coarse segmentation using full convolutional network, needs to be trained full convolutional network, can just obtain can be with Full convolutional network model for being split to RGB image.Wherein, in training process end to end, input can be passed through Data set is the image for being labelled with the binocular camera shooting of plane, and when mark needs to carry out the mark of pixel scale, multiple planes Content is indicated using multiple labels (label).Following S21~the S23 of specific training process:

S21, by the RGB image of the RGB image (referred to as A figure) of the left camera shooting of binocular camera and the shooting of right camera The mark of (referred to as B figure) progress pixel scale.Mark schematic diagram as shown in FIG. 6, what wherein pixel scale marked out is to ride certainly The people of driving.

S22, the A figure after mark and the B figure after mark are separately input into full convolutional network.

S23 obtains the available parameter of one group of full convolutional network using the mode training of the full convolutional network of general training, Available parameter is the model structure parameter of full convolutional network to get to trained full convolutional network model.

It is carried out before electronic equipment factory it should be noted that above-mentioned training process S21~S23 can be, also Say that trained full convolutional network model has been pre-configured in the electronic device before factory, then needing to make in user When being detected with the plane monitoring-network method to plane, the processor in electronic equipment can call directly the full volume of pre-configuration Product network model, is split processing to RGB image to be processed.Above-mentioned full convolutional network model is also possible in electronic equipment After factory, it is trained by above-mentioned training process S21~S23.

It, will when needing to be split using full convolutional network model RGB image to be processed collected to binocular camera RGB image to be processed is input in full convolutional network model, output result be include pixel scale mark semantic segmentation figure Picture.

In alternatively possible partitioning scheme, using convolutional neural networks (region-convolutional neural Networks, R-CNN) perhaps YOLO (you only look once) or SSD (single shot detector) etc. A variety of quick semantic segmentation modes, are split processing to RGB image to be processed.

Stating several R-CNN, YOLO, SSD etc. in use, quickly semantic segmentation mode carries out slightly RGB image to be processed Before segmentation, needs to be trained R-CNN model, YOLO model or SSD model, can just obtain can be used for RGB image R-CNN model, YOLO model or the SSD model being split.

In training process end to end, the data set of R-CNN model, YOLO model or SSD model is input to as mark The RGB image of plane, when mark, are marked by the way of bounding box, and the result obtained in this way is relative to pixel scale It is a thicker result for mark.Later, using general VGG-16 AlexNet mode training network.Specific instruction Practice the following S24~S25 of process:

S24, by the RGB image of the RGB image (referred to as A figure) of the left camera shooting of binocular camera and the shooting of right camera The mark of (referred to as B figure) progress pixel scale.Mark schematic diagram as shown in Figure 7, wherein the mode of bounding box marks Out be image-region that rectangle frame is outlined, as Fig. 7 rectangle frame in the object that is outlined be a vehicle.

Data set (B after the A after multiple groups mark schemes and marks schemes) is input to appointing in R-CNN, YOLO, SSD by S25 One neural network, training obtains corresponding network model, for example data set input R-CNN training is obtained corresponding R-CNN mould Type, YOLO correspond to YOLO model, SSD corresponds to SSD model.

It is carried out before electronic equipment factory it should be noted that training process S24~S25 can be, that is, Say that trained full convolutional network model has been pre-configured in the electronic device before factory, then user need using When the plane monitoring-network method detects plane, the processor in electronic equipment can call directly the full convolution of pre-configuration Network model is split processing to RGB image to be processed.Above-mentioned full convolutional network model is also possible to go out in electronic equipment After factory, it is trained by above-mentioned training process S21~S23.

The corresponding region of callout box that above two semantic segmentation mode obtains is carried out by granularity of a target object Segmentation, and a target object may include multiple planes, such as vehicle, in three dimensions corresponding cloud and more than one A plane, the result obtained in this way are simultaneously inaccurate.So each callout box obtained for above two semantic segmentation is corresponding Region, continue to divide, segmentation granularity can be configured according to actual needs, herein with no restrictions.

The partitioning scheme that being split for the corresponding region of each callout box may refer in above-described embodiment one carries out, Subsequent processing mode (such as clustering processing, etching operation, region growing operation) can also be with reference to related in above-described embodiment one Content, details are not described herein again.

In another partitioning scheme, image pre-segmentation processing is to be schemed using image pre-segmentation algorithm to RGB to be processed Segmentation as carrying out similar face block.Wherein image pre-segmentation algorithm is the algorithm clustered based on similarity between pixel, is obtained To segmentation result, RGB image as shown in Figure 8 a is schemed after obtaining segmentation as shown in Figure 8 b after image pre-segmentation algorithm Picture.Partitioning scheme compared to CNN, the segmentation result based on image pre-segmentation algorithm are more fine.

A figure and B figure are obtained a disparity map, then the disparity map that will be obtained by binocular parallax matching algorithm by step S3 Depth image is converted to, includes multiple subgraphs in the depth image.

In one example, the segmentation result based on above-mentioned steps S2, can scheme A and B figure is matched using binocular parallax Algorithm is handled, to realize the position of the face block obtained after reduction segmentation in three dimensions.Binocular parallax matching algorithm Basic principle is the position for searching for the current pixel of a wherein mesh camera in another mesh camera, passes through image coordinate, combines The position of the inside and outside ginseng also original pixel of binocular camera in space, therefore when carrying out binocular camera disparity computation, we can be with Some pixel is first extracted in a divisional plane block, polar curve search is carried out, when the points of search are more and more, so that it may Obtain many three-dimensional space points.Binocular parallax matching algorithm may include currently used BM, SGBM.

It should be understood that due to the subgraph after having been divided in above-mentioned steps S2 binocular vision can also be being carried out When difference matching, directly it may be the calculating of the subgraph progress binocular parallax of plane based on pre-segmentation result, and be reduced directly The three-dimensional space position of these subgraphs.Calculation amount can be reduced in this way, accelerate the speed of plane monitoring-network.

When semantic segmentation obtains subgraph, it is flat can to determine whether the target object split has according to mark Face only can determine disparity map to the subgraph with face block then when determining disparity map according to step S3, again will later Obtained disparity map is converted to depth image.Can reduce the treating capacity of subsequent cluster in this way, save detection planarization process when Between.

Step S4 determines the son of corresponding cloud in the same plane from each subgraph that depth image includes Image.

Herein, plane fitting can be carried out to each subgraph by PCA algorithm, calculates the mean square error of fit Plane Difference, if the mean square error of fit Plane is less than first threshold, then corresponding cloud of the subgraph is in the same plane.

Step S5 carries out clustering processing, obtains to the subgraph of each corresponding cloud determined in the same plane To multiple coarse extraction planes.

In step s 5, clustering processing process can refer to the phase of the clustering processing process in above-described embodiment one inside the Pass Hold, details are not described herein again.

Step S6 carries out process of refinement, plane after being optimized to each coarse extraction plane.

In step s 6, one of erosion algorithm, algorithm of region growing and merging plane operations or more can be used To coarse extraction plane process of refinement obtained in step S5, correlated process can refer to the phase of above-described embodiment one inside the Pass for kind operation Hold.

It should be understood that after above-mentioned steps S3 obtains depth image, above-described embodiment can also be used in the embodiment of the present application Two processing mode, which is realized, detects multiple planes, and related content can refer to the related content of embodiment two, and details are not described herein again.

By three above embodiment, on the one hand may be implemented to detection while multiple planes in scene, and image Sensor is no longer limited to depth camera, and algorithm is also no longer limited to traditional clustering algorithm.On the other hand, by being carried out to image Pre-segmentation, then handling respectively the image after pre-segmentation can guarantee that the high speed of plane detection algorithm executes, thus The multiple planes being quickly detected in scene.

A kind of plane monitoring-network method is also provided in some other embodiment, in the embodiment of the present application, is set applied to calculating It is standby, referring to Fig. 9, method includes the following steps:

Step 901, image data to be processed is obtained.

Step 902, semantic segmentation is carried out to image data to be processed, obtains N number of sub-image data with markup information, N is the integer greater than 1；Markup information is used to mark the target object in sub-image data.

Step 903, according to the markup information of each sub-image data, from N number of sub-image data with markup information Determine the Q sub-image datas with plane；Q is the integer greater than 0 and less than or equal to N.

Step 904, determine that each the sub-image data with plane is corresponding in the Q sub-image datas with plane Point cloud information.

Step 905, according to the sub-image data corresponding point each in the Q sub-image datas with plane with plane Cloud information determines K coarse extraction plane from the Q sub-image datas with plane；Wherein, K is whole more than or equal to Q Number.

Step 906, processing is optimized to K coarse extraction plane, obtains plane after L optimization；L is just no more than K Integer.

In one possible implementation, the image data to be processed in above-mentioned steps 901 includes being clapped by binocular camera The first RGB image and the second RGB image taken the photograph.

Above-mentioned steps 902 may be accomplished by: carrying out semantic segmentation to first RGB image, obtains N number of tool There is the first subgraph of markup information；And semantic segmentation is carried out to second RGB image, it obtains N number of with markup information Second subgraph；Wherein, first subgraph with markup information and with first subgraph with position The second subgraph with markup information of corresponding relationship forms the sub-image data with markup information.

Illustratively, the first RGB image can be the A figure in embodiment three, and the second RGB image can be in embodiment three B figure.

Above-mentioned steps 904 may be accomplished by: for the Q each of the sub-image datas with plane Sub-image data with plane executes: having markup information according to include in the sub-image data with plane First subgraph and the second subgraph with markup information with first subgraph with position corresponding relationship, really Make disparity map；Sub- depth image is determined according to the disparity map；According to the sub- depth image, determine described with flat The corresponding point cloud information of the sub-image data in face.

Above-mentioned steps 905 may be accomplished by: according to corresponding cloud of the sub-image data with plane Information determines K coarse extraction plane from Q sub- depth images.

It should be noted that determining K coarse extraction plane and clustering processing and being optimized to coarse extraction plane Etc. processes specific implementation may refer to the related embodiment in embodiment one description, details are not described herein again.

Above-mentioned plane monitoring-network method can be also used for intelligent robot navigation in addition to being applied to augmented reality field, from It is dynamic the fields such as to drive, so as to realize barrier avoiding function that computer judges automatically.

In other embodiments of the application, the embodiment of the present application also provides a kind of calculating equipment, as shown in Figure 10, should Calculating equipment may include: processor 1001；Memory 1002；And one or more computer programs 1003, above-mentioned each device Part can be connected by one or more communication bus 1004.

Wherein the one or more computer program 1003 is stored in above-mentioned memory 1002 and is configured as being located It manages device 1001 to execute, which includes instruction, and exemplary, above-metioned instruction can be used for executing such as Each step in Fig. 2 and Fig. 5 in corresponding embodiment.Specifically, processor 1001 can be used for executing the step in Fig. 2 201- step 205, processor 1001 are used to execute the step S1- step S6 in Fig. 5.

Through the above description of the embodiments, it is apparent to those skilled in the art that, for description It is convenienct and succinct, only the example of the division of the above functional modules, in practical application, can according to need and will be upper It states function distribution to be completed by different functional modules, i.e., the internal structure of device is divided into different functional modules, to complete All or part of function described above.The specific work process of the system, apparatus, and unit of foregoing description, before can referring to The corresponding process in embodiment of the method is stated, details are not described herein.

Each functional unit in each embodiment of the embodiment of the present application can integrate in one processing unit, can also be with It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list Member both can take the form of hardware realization, can also realize in the form of software functional units.

If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product When, it can store in a computer readable storage medium.Based on this understanding, the technical solution of the embodiment of the present application Substantially all or part of the part that contributes to existing technology or the technical solution can be with software product in other words Form embody, which is stored in a storage medium, including some instructions use so that one It is each that computer equipment (can be personal computer, the first electronic equipment or the network equipment etc.) or processor execute the application The all or part of the steps of a embodiment the method.And storage medium above-mentioned include: flash memory, mobile hard disk, only Read the various media that can store program code such as memory, random access memory, magnetic or disk.

The above, the only specific embodiment of the embodiment of the present application, but the protection scope of the embodiment of the present application is not It is confined to this, anyone skilled in the art can think easily in the technical scope that the embodiment of the present application discloses The change or replacement arrived should all be covered within the protection scope of the embodiment of the present application, therefore the protection model of the embodiment of the present application Enclosing should be subject to the protection scope in claims.

Claims

1. a kind of plane monitoring-network method is applied to calculate equipment, which is characterized in that the described method includes:

Obtain image data to be processed；

The image data to be processed is split, N number of sub-image data is obtained, the N is the integer greater than 1；

Determine the corresponding point cloud information of at least one sub-image data in N number of sub-image data；

The corresponding point cloud information of at least one sub-image data according in N number of sub-image data, to N number of son Corresponding cloud of image data carries out clustering processing, obtains K coarse extraction plane；The K is the positive integer no more than the N；

Processing is optimized to the K coarse extraction plane, obtains plane after L optimization；The L is just no more than the K Integer.

2. the method as described in claim 1, which is characterized in that the image data to be processed is depth image；The depth It include the image coordinate and depth value of each pixel in image；

It is described that the image data to be processed is split, obtain N number of sub-image data, comprising:

The depth image is split, N number of sub- depth image is obtained；

The corresponding point cloud information of at least one sub-image data in determination N number of sub-image data, comprising:

Determine at least one of N number of sub- depth image corresponding point cloud information of sub- depth image；

The corresponding point cloud information of at least one sub-image data according in N number of sub-image data, to N number of son Corresponding cloud of image data carries out clustering processing, obtains K coarse extraction plane, comprising:

According to the corresponding point cloud information of every sub- depth image in N number of sub- depth image, every sub- depth image is determined The mean square error of plane after the fitting of corresponding cloud；

The sub- depth image for meeting first condition is determined from N number of sub- depth image, forms subgraph image set to be processed；Institute The mean square error for stating plane after the fitting that first condition includes the corresponding cloud of sub- depth image is less than or equal to the first threshold Value；

It concentrates the corresponding cloud of sub- depth image for including to cluster the subgraph to be processed, it is flat to obtain K coarse extraction Face.

3. the method as described in claim 1, which is characterized in that the image data to be processed is depth image；

The depth image is split, N number of sub- depth image is obtained；

Every corresponding cloud of sub- depth image in N number of sub- depth image is constructed into graph structure as a node；It is described Each node in graph structure is stored with the corresponding point cloud information of the node；

Each node in the graph structure is traversed, determines two nodes for meeting second condition in the graph structure, then Side is constructed between two nodes for meeting the second condition；The second condition is corresponding including any node in two nodes Point cloud depth value is continuous and the normal vector of two node corresponding points clouds between angle be less than angle threshold value；

Determine sub- depth map corresponding with having the node at least one side in the graph structure in N number of sub- depth image Picture forms subgraph image set to be processed；

4. the method as described in claim 1, which is characterized in that the image data to be processed includes being shot by binocular camera First RGB image and the second RGB image；

Image pre-segmentation is carried out to first RGB image, obtains N number of first face block；And figure is carried out to second RGB image As pre-segmentation, N number of second face block is obtained；First face block and second face block have position corresponding relationship；

For each first face block in N number of first face block, execute:

Second face block that there is position corresponding relationship with first face block is determined from N number of second face block；

There is second face block of position corresponding relationship according to first face block and with first face block, determine Disparity map；Sub- depth image is determined according to the disparity map；

Subgraph image set to be processed is formed according to the N number of sub- depth image determined；

The corresponding point cloud information of at least one sub-image data according in N number of sub-image data, to the N Corresponding cloud of a sub-image data carries out clustering processing, obtains K coarse extraction plane, comprising:

The corresponding point cloud information of at least one sub-image data according in N number of sub-image data, to described wait locate Reason subgraph concentrates the N number of corresponding cloud of sub- depth image for including to be clustered, and obtains K coarse extraction plane.

5. such as the described in any item methods of claim 2-4, which is characterized in that described to include to the subgraph concentration to be processed Corresponding cloud of sub- depth image clustered, obtain K coarse extraction plane, comprising:

The corresponding point cloud information of the sub- depth image of every for including is concentrated according to the subgraph to be processed, establishes minimum heap data Structure；The minimum heap data structure is used for the mean square error according to every sub- corresponding cloud of depth image to described to be processed The sub- depth image that subgraph is concentrated is ranked up, and most positioned at the mean square error of the corresponding cloud of sub- depth image on heap top It is small；

For the minimum heap data structure, predetermined registration operation is executed, until any two section in the minimum heap data structure The mean square error of plane is greater than the first threshold after the fitting of corresponding cloud of point, obtains the K coarse extraction plane；

Wherein, the predetermined registration operation include: from taking out sub- depth image in heap top in the minimum heap data structure, if from institute It states in the adjacent sub- depth image of sub- depth image and determines the sub- depth image for meeting third condition, then by the sub- depth map As being merged with the sub- depth image for meeting third condition, sub- depth image after being merged, the third condition packet Mean square error after including point-cloud fitting plane corresponding with the sub- depth image is less than first threshold and the mean square error most It is small；Sub- depth image after the fusion is added into the minimum heap data structure.

6. the method as described in claim 1, which is characterized in that the image data to be processed is the point for including in three-dimensional space Cloud；

Using the three-dimensional space as the node of the first level of octree structure；

For the first level and the i-th level each child node for including in the octree structure, execute: if the sub- section Point meets fourth condition, then carries out eight equal portions segmentations to the child node, obtain eight child nodes of i+1 level；Wherein institute It states the mean square error that fourth condition includes corresponding cloud of child node and is greater than the first threshold；The i is the integer greater than 1, I-th level includes 8i child node；

Until all child nodes for including in last described level meet fifth condition, building obtain include M level son The octree structure of node；The fifth condition includes the mean square error of corresponding cloud of child node no more than first threshold Value, alternatively, corresponding cloud of child node includes that point quantity is less than amount threshold；

Determine N number of undivided child node in the octree structure；

Determine the undivided corresponding point cloud information of child node of at least one of described N number of undivided child node；

The corresponding point cloud information of at least one sub-image data according in N number of sub-image data is described right Corresponding cloud of N number of sub-image data carries out clustering processing, obtains K coarse extraction plane, comprising:

According to the undivided corresponding point cloud information of child node of at least one of N number of undivided child node, to described Corresponding cloud of N number of undivided child node described in octree structure carries out clustering processing, obtains K coarse extraction plane.

7. method as claimed in claim 6, which is characterized in that it is described according in N number of undivided child node at least The corresponding point cloud information of one undivided child node, it is corresponding to N number of undivided child node described in the octree structure Point cloud carry out clustering processing, obtain K coarse extraction plane, comprising:

According to the undivided corresponding point cloud information of child node of at least one of N number of undivided child node, institute is determined The normal vector of each corresponding cloud of undivided child node in N number of undivided child node is stated, and the normal vector is passed through Hough transformation is at the point in parameter space；

Determine the K point set that the normal vector of N number of corresponding cloud of undivided child node is formed in parameter space, often A point set has an aggregation center；

For each point set, the point fallen in the pericentral preset range of aggregation of the point set is determined；

The permeate coarse extraction of the corresponding corresponding cloud of undivided child node of the point fallen within a preset range is put down Face.

8. the method according to claim 1 to 7, which is characterized in that described excellent to K coarse extraction plane progress Change processing obtains plane after L optimization, comprising:

Determine the normal vector of each coarse extraction plane in the K coarse extraction plane；

Any coarse extraction plane in the K coarse extraction plane is traversed, meets the coarse extraction plane of Article 6 part if it exists, then The coarse extraction plane is permeated a plane with the coarse extraction plane for meeting the Article 6 part, is obtained flat after L optimization Face；

Wherein, the Article 6 part includes: that normal vector is parallel with the normal vector of the coarse extraction plane and flat with the coarse extraction Variance after the fit Plane of face is less than variance threshold values.

9. a kind of plane monitoring-network method is applied to calculate equipment, which is characterized in that the described method includes:

Obtain image data to be processed；

Semantic segmentation is carried out to the image data to be processed, obtains N number of sub-image data with markup information, the N is Integer greater than 1；The markup information is used to mark the target object in the sub-image data；

According to the markup information of each sub-image data, Q are determined from N number of sub-image data with markup information Sub-image data with plane；The Q is the integer greater than 0 and less than or equal to the N；

Determine that each corresponding cloud of the sub-image data with plane is believed in the Q sub-image datas with plane Breath；

According to the sub-image data corresponding point cloud information each in the Q sub-image datas with plane with plane, K coarse extraction plane is determined from the Q sub-image datas with plane；The K is whole more than or equal to the Q Number；

10. method as claimed in claim 9, which is characterized in that the image data to be processed includes being shot by binocular camera The first RGB image and the second RGB image；

It is described that semantic segmentation is carried out to the image data to be processed, N number of sub-image data with markup information is obtained, is wrapped It includes:

Semantic segmentation is carried out to first RGB image, obtains N number of the first subgraph with markup information；And to described Two RGB images carry out semantic segmentation, obtain N number of the second subgraph with markup information；Wherein, each described with mark letter The first subgraph ceased and the second subgraph with markup information with first subgraph with position corresponding relationship One sub-image data with markup information of composition；

It is described to determine in the Q sub-image datas with plane each with corresponding cloud of sub-image data of plane Information, comprising:

For sub-image data of each of the Q sub-image datas with plane with plane, execute:

According to including the first subgraph with markup information and with described in the sub-image data with plane Second subgraph with markup information of one subgraph with position corresponding relationship, determines disparity map；

Sub- depth image is determined according to the disparity map；

According to the sub- depth image, the corresponding point cloud information of sub-image data with plane is determined；

It is described according in a sub-image datas with plane of the Q each to there is corresponding cloud of sub-image data of plane to believe Breath determines K coarse extraction plane from the Q sub-image datas with plane, comprising:

According to the corresponding point cloud information of the sub-image data with plane, K coarse extraction is determined from Q sub- depth images Plane.

11. the method as described in claim 9 or 10, which is characterized in that the sub-image data according to plane Corresponding point cloud information determines K coarse extraction plane from Q sub- depth images, comprising:

Every corresponding cloud of sub- depth image in the Q sub- depth images is constructed into graph structure as a node；It is described Each node in graph structure is stored with the corresponding point cloud information of the node；

Determine sub- depth map corresponding with having the node at least one side in the graph structure in the Q sub- depth images Picture forms subgraph image set to be processed；

It concentrates the corresponding cloud of sub- depth image for including to carry out clustering processing the subgraph to be processed, obtains K and slightly mention It makes even face.

12. method as claimed in claim 11, which is characterized in that described to concentrate the son for including deep the subgraph to be processed Degree corresponding cloud of image is clustered, and K coarse extraction plane is obtained, comprising:

Wherein, the predetermined registration operation include: from taking out sub- depth image in heap top in the minimum heap data structure, if from institute It states in the adjacent sub- depth image of sub- depth image and determines the sub- depth image for meeting third condition, then by the sub- depth map As being merged with the sub- depth image for meeting third condition, sub- depth image after being merged；The third condition packet Mean square error after including point-cloud fitting plane corresponding with the sub- depth image is less than first threshold and the mean square error most It is small；Sub- depth image after the fusion is added into the minimum heap data structure.

13. such as the described in any item methods of claim 9-12, which is characterized in that described to be carried out to the K coarse extraction plane Optimization processing obtains plane after L optimization, comprising:

Any coarse extraction plane in the K coarse extraction plane is traversed, meets the coarse extraction plane of Article 6 part if it exists, then The coarse extraction plane is permeated a plane with the coarse extraction plane for meeting the Article 6 part；

14. a kind of calculating equipment, which is characterized in that including at least one processor；

At least one described processor, is configured as performing the following operations:

Obtain image data to be processed；

15. calculating equipment as claimed in claim 14, which is characterized in that the image data to be processed is depth image；Institute State the image coordinate and depth value in depth image including each pixel；

The depth image is split, N number of sub- depth image is obtained；

16. calculating equipment as claimed in claim 14, which is characterized in that the image data to be processed is depth image；

The depth image is split, N number of sub- depth image is obtained；

17. calculating equipment as claimed in claim 14, which is characterized in that the image data to be processed includes by binocular camera The first RGB image and the second RGB image of shooting；

For each first face block in N number of first face block, execute:

18. such as the described in any item calculating equipment of claim 15-17, which is characterized in that described to the subgraph to be processed Corresponding cloud of sub- depth image that concentration includes is clustered, and K coarse extraction plane is obtained, comprising:

19. calculating equipment as claimed in claim 14, which is characterized in that the image data to be processed is to wrap in three-dimensional space The point cloud included；

Determine N number of undivided child node in the octree structure；

The corresponding point cloud information of at least one sub-image data in determination N number of sub-image data；

20. calculating equipment as claimed in claim 19, which is characterized in that described according in N number of undivided child node The corresponding point cloud information of at least one undivided child node, N number of undivided son described in the octree structure is saved Corresponding cloud of point carries out clustering processing, obtains K coarse extraction plane, comprising:

21. such as the described in any item calculating equipment of claim 14-20, which is characterized in that described to the K coarse extraction plane Processing is optimized, plane after L optimization is obtained, comprising:

22. a kind of calculating equipment, which is characterized in that including at least one processor；

Obtain image data to be processed；

23. calculating equipment as claimed in claim 22, which is characterized in that the image data to be processed includes by binocular camera The first RGB image and the second RGB image of shooting；

It is described that semantic segmentation is carried out to the image data to be processed, N number of sub-image data with markup information is obtained, it is described N is the integer greater than 1, comprising:

Sub- depth image is determined according to the disparity map；

24. the calculating equipment as described in claim 22 or 23, which is characterized in that the subgraph according to plane The corresponding point cloud information of data determines K coarse extraction plane from Q sub- depth images, comprising:

25. calculating equipment as claimed in claim 24, which is characterized in that described to include to the subgraph concentration to be processed Corresponding cloud of sub- depth image is clustered, and K coarse extraction plane is obtained, comprising:

26. such as the described in any item calculating equipment of claim 22-25, which is characterized in that described to the K coarse extraction plane Processing is optimized, plane after L optimization is obtained, comprising:

27. a kind of circuit system, which is characterized in that including at least one processing circuit, be configured as executing as claim 1 to 8 any methods, alternatively, executing the method as described in claim 9 to 13 is any.

28. a kind of computer storage medium, which is characterized in that the computer storage medium includes computer program, works as calculating When machine program is run on the computing device, so that the calculating equipment executes method as described in any of the claims 1 to 8, or Person executes the method as described in claim 9 to 13 is any.