CN108629799B - Method and equipment for realizing augmented reality - Google Patents

Method and equipment for realizing augmented reality Download PDF

Info

Publication number
CN108629799B
CN108629799B CN201710181831.7A CN201710181831A CN108629799B CN 108629799 B CN108629799 B CN 108629799B CN 201710181831 A CN201710181831 A CN 201710181831A CN 108629799 B CN108629799 B CN 108629799B
Authority
CN
China
Prior art keywords
plane
scene image
frame
environmental scene
triangular
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710181831.7A
Other languages
Chinese (zh)
Other versions
CN108629799A (en
Inventor
陈卓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Idealsee Technology Co Ltd
Original Assignee
Chengdu Idealsee Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Idealsee Technology Co Ltd filed Critical Chengdu Idealsee Technology Co Ltd
Priority to CN201710181831.7A priority Critical patent/CN108629799B/en
Publication of CN108629799A publication Critical patent/CN108629799A/en
Application granted granted Critical
Publication of CN108629799B publication Critical patent/CN108629799B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38Registration of image sequences
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4038Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10028Range image; Depth image; 3D point clouds

Abstract

The invention discloses a method and equipment for realizing augmented reality, wherein a certain frame of environmental scene image is selected from environmental scene images collected in real time as a1 st frame of environmental scene image, a Slam algorithm is utilized to carry out image registration on adjacent frames of environmental scene images, and the 1 st frame of environmental scene image and three-dimensional point cloud data of each frame of image behind the 1 st frame of environmental scene image are calculated; acquiring newly-added three-dimensional point cloud data of the ith frame relative to the (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set; splicing each newly added triangular surface in the newly added triangular surface set into the plane set of the (i-1) th frame environmental scene image to obtain the plane set of the i th frame environmental scene image; and selecting a visible plane from the plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and overlapping the virtual information to be displayed in the selected target plane for displaying.

Description

Method and equipment for realizing augmented reality
Technical Field
The invention relates to the technical field of augmented reality, in particular to a method and equipment for realizing augmented reality.
Background
Augmented Reality (AR) generates virtual objects that do not exist in the real environment by means of computer graphics and visualization, and accurately fuses the virtual objects into the real environment by means of image recognition and positioning technology, and integrates the virtual objects with the real environment by means of a display device, and presents the real sensory experience to the user.
The existing AR-Glass equipment obtains scene 3D information (three-dimensional reconstruction) and the posture (position and rotation data) of the equipment in a scene according to data of an input end (one or more of a gray camera, a color camera, a depth camera, sensor equipment and the like) through an SLAM algorithm, and presents a real scene in a video segment-through or optical segment-through mode. And simultaneously, virtual information (characters, pictures, videos, three-dimensional models, independent three-dimensional scenes and the like) is fused and displayed in the real scene according to the scene 3D information obtained by the SLAM algorithm and the posture of the equipment in the scene. However, when the virtual information is displayed, the posture of the virtual information in a real scene is usually preset or the user manually controls the virtual information to move to a position to be displayed, and the mode is complex to operate and has the problem of large deviation in virtual-real fusion.
Disclosure of Invention
The invention provides a method and equipment for realizing augmented reality, which can automatically control virtual information to move to a position to be displayed, simplify operation and effectively reduce deviation existing in virtual-real fusion.
An embodiment of the present application provides a method for implementing augmented reality, including:
selecting a certain frame of environmental scene image from the environmental scene images collected in real time as a1 st frame of environmental scene image, performing image registration on adjacent frames of environmental scene images by using a Slam algorithm, and calculating the 1 st frame of environmental scene image and three-dimensional point cloud data of each frame of environmental scene image;
sequentially executing the following steps for the 1 st frame environmental scene image and each frame environmental scene image after the 1 st frame environmental scene image:
acquiring newly-added three-dimensional point cloud data of an ith frame relative to an (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set, wherein i is sequentially taken from 1 to n, and n is an integer not less than 2;
splicing each newly added triangular surface in the newly added triangular surface set into the plane set of the (i-1) th frame environment scene image to obtain a plane set of the i th frame environment scene image, wherein the plane set comprises a vertex set and a normal direction of each plane;
and selecting a visible plane from the plane set of the ith frame of environment scene image as a target plane based on a preset condition, and superposing virtual information to be displayed on the selected target plane for displaying, wherein whether the plane is the visible plane is judged based on the normal direction of each plane.
Optionally, the acquiring newly added three-dimensional point cloud data of the ith frame relative to the (i-1) th frame environmental scene image specifically includes:
and comparing the calculated three-dimensional point cloud data of the ith frame of environmental scene image with the three-dimensional point cloud data of the (i-1) th frame of environmental scene image to obtain the newly-added three-dimensional point cloud data.
Optionally, when i is equal to 1, the acquiring new three-dimensional point cloud data of the ith frame relative to the (i-1) th frame environmental scene image, and processing the new three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed new triangular surface set specifically includes:
taking the three-dimensional point cloud data of the 1 st frame of environmental scene image as newly-added three-dimensional point cloud data of the 1 st frame of environmental scene image;
and processing the three-dimensional point cloud data of the 1 st frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a newly-added triangular surface set of the 1 st frame of environmental scene image formed by all the processed triangular surfaces.
Optionally, when i is equal to 1, the splicing each newly added triangular surface in the newly added triangular surface set to the plane set of the (i-1) th frame environmental scene image to obtain the plane set of the i th frame environmental scene image specifically includes:
and performing triangular surface splicing on the newly added triangular surface set of the 1 st frame environmental scene image by taking a certain three-dimensional point in the 1 st frame environmental scene image as a reference, and forming a plane set of the 1 st frame environmental scene image by all planes obtained by splicing.
Optionally, the splicing each triangular surface in the newly added triangular surface set to the plane set of the (i-1) th frame environmental scene image to obtain the plane set of the i th frame environmental scene image specifically includes:
step 1: determining all planes adjacent to the newly added triangular surface in the newly added triangular surface set from the plane set of the ith frame of environmental scene image to form a boundary plane;
step 2: expanding the boundary plane towards the direction of the adjacent newly-added triangular surface, and calculating the normal relation between the adjacent newly-added triangular surface and the boundary plane;
and step 3: judging whether the adjacent newly-added triangular surfaces and the boundary plane are coplanar or not by utilizing the calculated normal relation; if the newly increased triangular surfaces are coplanar, splicing the adjacent newly increased triangular surfaces into the boundary plane, and updating the boundary information of the boundary plane; if not, taking the adjacent newly-added triangular surface as a new plane, and adding the adjacent newly-added triangular surface into the boundary plane, wherein the boundary information of each plane is the outermost vertex set and the normal direction in the triangular surface set forming the plane;
and (5) repeatedly executing the step (1) to the step (3) until all the newly added triangular surfaces in the newly added triangular surface set are calculated, and taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image.
Optionally, the taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image specifically includes:
detecting whether the area of each plane in the calculated plane set is not smaller than a first preset area or not;
and when a plane smaller than the first preset area is detected, splicing the plane smaller than the first preset area into an adjacent plane which is most matched with the normal of the plane according to the normal relation until all planes smaller than the first preset area in the calculated plane set are spliced into the adjacent plane, so as to obtain the plane set of the ith frame of environmental scene image.
Optionally, based on the preset condition, selecting a visible plane from the plane set of the ith frame of environmental scene image as a target plane, and superimposing the virtual information to be displayed on the selected target plane for displaying, specifically:
collecting a throwing track for throwing the virtual information;
and acquiring a visible plane closest to the throwing track from the plane set of the ith frame environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Optionally, when a visible plane closest to the throwing track is obtained from the plane set of the ith frame environment scene image as the target plane, the method further includes:
judging whether the area of the nearest visible plane is not smaller than the occupied area for displaying the virtual information;
if the area of the nearest visible plane is judged to be not smaller than the occupied area, the nearest visible plane is used as the target plane, and the virtual information is superposed in the target plane to be displayed;
and if the area of the nearest visible plane is judged to be smaller than the occupied area, selecting one visible plane larger than the occupied area from the plane set of the ith frame of environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Optionally, after obtaining the plane set of the ith frame of environmental scene image, the method further includes:
detecting whether the plane set of the ith frame environment scene image is updated compared with the plane set of the (i-1) th frame environment scene image;
if not, processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm, and forming a triangular surface set of the ith frame of environmental scene image by using all triangular surfaces obtained by processing;
and performing triangular surface splicing by taking a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and forming a plane set of the ith frame of environmental scene image by all planes obtained by splicing.
Optionally, after obtaining the plane set of the ith frame of environmental scene image, the method further includes:
and marking each plane in the plane set of the ith frame of environmental scene image in an indexing mode.
Optionally, after obtaining the plane set of the ith frame of environmental scene image, the method further includes:
detecting whether the sum of the areas of newly added triangular surfaces in each plane in the plane set of the ith frame of environmental scene image is not smaller than a second preset area;
when the sum of the areas of newly added triangular surfaces in a certain plane is detected to be smaller than the second preset area, processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a triangular surface set of the ith frame of environmental scene image;
performing triangle surface splicing by taking a certain triangle surface in the triangle surface set of the ith frame of environment scene image as a reference, and forming a new plane set of the ith frame of environment scene image by all planes obtained by splicing;
and selecting a visible plane from the new plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and overlapping the virtual information to be displayed in the selected target plane for displaying.
Another embodiment of the present application further provides an apparatus for implementing augmented reality, including:
the system comprises a three-dimensional point cloud data acquisition unit, a processing unit and a processing unit, wherein the three-dimensional point cloud data acquisition unit is used for selecting a certain frame of environmental scene image from environmental scene images acquired in real time as a1 st frame of environmental scene image, carrying out image registration on adjacent frames of environmental scene images by using a Slam algorithm, and calculating the 1 st frame of environmental scene image and three-dimensional point cloud data of each frame of environmental scene image behind the 1 st frame of environmental scene image;
a loop execution unit, configured to sequentially execute the following steps for the 1 st frame of environmental scene image and each frame of environmental scene image thereafter: acquiring newly-added three-dimensional point cloud data of an ith frame relative to an (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set, wherein i is sequentially taken from 1 to n, and n is an integer not less than 2; splicing each newly added triangular surface in the newly added triangular surface set into the plane set of the (i-1) th frame environment scene image to obtain a plane set of the i th frame environment scene image, wherein the plane set comprises a vertex set and a normal direction of each plane; and selecting a visible plane from the plane set of the ith frame of environment scene image as a target plane based on a preset condition, and superposing virtual information to be displayed on the selected target plane for displaying, wherein whether the plane is the visible plane is judged based on the normal direction of each plane.
Optionally, the loop execution unit further includes:
and the newly added three-dimensional point cloud data acquisition unit is used for comparing the calculated three-dimensional point cloud data of the ith frame of environmental scene image with the three-dimensional point cloud data of the (i-1) th frame of environmental scene image to obtain the newly added three-dimensional point cloud data.
Optionally, the loop execution unit further includes:
a newly-added triangular surface set acquisition unit, configured to use the three-dimensional point cloud data of the 1 st frame of environmental scene image as newly-added three-dimensional point cloud data of the 1 st frame of environmental scene image when i is equal to 1; and processing the three-dimensional point cloud data of the 1 st frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a newly-added triangular surface set of the 1 st frame of environmental scene image formed by all the processed triangular surfaces.
Optionally, the loop execution unit further includes:
and the plane set acquisition unit is used for performing triangular plane splicing on the newly added triangular plane set of the 1 st frame environmental scene image by taking a certain three-dimensional point in the 1 st frame environmental scene image as a reference when i is equal to 1, and forming the plane set of the 1 st frame environmental scene image by all planes obtained through splicing.
Optionally, the plane set obtaining unit is further configured to determine, from the plane set of the ith frame of environmental scene image, all planes adjacent to the newly added triangular surface in the newly added triangular surface set to form a boundary plane; expanding the boundary plane towards the direction of the adjacent newly-added triangular surface, and calculating the normal relation between the adjacent newly-added triangular surface and the boundary plane; judging whether the adjacent newly-added triangular surfaces and the boundary plane are coplanar or not by utilizing the calculated normal relation; if the newly increased triangular surfaces are coplanar, splicing the adjacent newly increased triangular surfaces into the boundary plane, and updating the boundary information of the boundary plane; if not, taking the adjacent newly-added triangular surface as a new plane, and adding the adjacent newly-added triangular surface into the boundary plane, wherein the boundary information of each plane is the outermost vertex set and the normal direction in the triangular surface set forming the plane; and repeatedly executing the steps until all the newly-added triangular surfaces in the newly-added triangular surface set are calculated, and taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image.
Optionally, the plane set obtaining unit is further configured to detect whether an area of each plane in the calculated plane set is not smaller than a first preset area; and when a plane smaller than the first preset area is detected, splicing the plane smaller than the first preset area into an adjacent plane which is most matched with the normal of the plane according to the normal relation until all planes smaller than the first preset area in the calculated plane set are spliced into the adjacent plane, so as to obtain the plane set of the ith frame of environmental scene image.
Optionally, the loop execution unit further includes:
the superposition display unit is used for acquiring a throwing track for throwing the virtual information; and acquiring a visible plane closest to the throwing track from the plane set of the ith frame environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Optionally, the apparatus further comprises:
an area determination unit, configured to determine whether an area of a nearest visible plane closest to the throwing trajectory is not smaller than an occupied area for displaying the virtual information when the visible plane closest to the throwing trajectory is acquired from the plane set of the ith frame of environmental scene image as the target plane;
the superposition display unit is used for taking the nearest visible plane as the target plane and superposing the virtual information in the target plane for display when the area of the nearest visible plane is judged to be not smaller than the occupied area; and when the area of the nearest visible plane is judged to be smaller than the occupied area, selecting one visible plane larger than the occupied area from the plane set of the ith frame of environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Optionally, the apparatus further comprises:
a plane set update detection unit, configured to detect whether the plane set of the ith frame environmental scene image is updated compared with the plane set of the (i-1) th frame environmental scene image after obtaining the plane set of the ith frame environmental scene image;
the triangular surface set acquisition unit is used for processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm when the situation that the image is not updated is detected, and forming a triangular surface set of the ith frame of environmental scene image by using all triangular surfaces obtained by processing;
the plane set obtaining unit is further configured to perform triangular surface stitching with a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and form a plane set of the ith frame of environmental scene image with all planes obtained through stitching.
Optionally, the apparatus further comprises:
and the index marking unit is used for marking each plane in the plane set of the ith frame environment scene image in an index mode after the plane set of the ith frame environment scene image is obtained.
Optionally, the apparatus further comprises:
the triangular surface area detection unit is used for detecting whether the sum of the areas of newly added triangular surfaces in each plane in the plane set of the ith frame of environment scene image is not less than a second preset area after the plane set of the ith frame of environment scene image is obtained;
the triangular surface set obtaining unit is further configured to, when it is detected that the sum of the areas of newly added triangular surfaces in a certain plane is smaller than the second preset area, process the three-dimensional point cloud data of the i-th frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a triangular surface set of the i-th frame of environmental scene image;
the plane set acquisition unit is further configured to perform triangular surface stitching with a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and form a new plane set of the ith frame of environmental scene image with all planes obtained by stitching;
and the superposition display unit is also used for selecting a visible plane from the new plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and superposing the virtual information to be displayed on the selected target plane for displaying.
The invention has the following beneficial effects:
the method comprises the steps of processing newly-added three-dimensional point cloud data of two adjacent frames of environmental scene images to obtain a plane set of each frame of environmental scene image; then based on preset conditions, selecting a visible plane from the plane set of each frame of the environment scene image as a target plane, and the virtual information to be displayed is superposed in the selected target plane for display, so that the embodiment of the application automatically calculates the target plane through equipment, then the virtual object is overlaid in the target plane for displaying, without the user manually operating the virtual object to move the virtual object to the target plane for displaying, thereby realizing the automatic control of the virtual object to move to the position to be displayed, and the accuracy of the target plane calculated by the device is obviously improved compared with the accuracy of manual movement, and need not user manual operation, so, realized simplifying operation and can effectively reduce the effect of the deviation that virtual reality fuses to exist.
Drawings
FIG. 1 is a flowchart of a method for implementing augmented reality according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for obtaining a planar set of an i-th frame of an environmental scene image according to an embodiment of the present invention;
fig. 3 is a block diagram of an implementation of an augmented reality device in an embodiment of the present invention.
Detailed Description
The invention provides a method and equipment for realizing augmented reality, which can automatically control virtual information to move to a position to be displayed, simplify operation and effectively reduce deviation existing in virtual-real fusion.
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the present invention discloses a method for realizing augmented reality, which specifically comprises the following steps:
s101, selecting a certain frame of environmental scene image from the environmental scene images collected in real time as a1 st frame of environmental scene image, performing image registration on adjacent frames of environmental scene images by using a Slam algorithm, and calculating the 1 st frame of environmental scene image and three-dimensional point cloud data of each frame of environmental scene image;
s102, sequentially executing the following steps for the 1 st frame of environment scene image and each frame of environment scene image after the 1 st frame of environment scene image:
s103, acquiring newly-added three-dimensional point cloud data of the ith frame relative to the (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set, wherein i is sequentially taken from 1 to n, and n is an integer not less than 2;
s104, splicing each newly added triangular surface in the newly added triangular surface set into a plane set of the (i-1) th frame environment scene image to obtain a plane set of the i th frame environment scene image, wherein the plane set comprises a vertex set and a normal direction of each plane;
s105, selecting a visible plane from the plane set of the ith frame of environment scene image as a target plane based on preset conditions, and superposing virtual information to be displayed on the selected target plane for displaying, wherein whether the plane is the visible plane is judged based on the normal direction of each plane.
In step S101, an environmental scene image may be acquired in real time by using an imaging device such as a camera, a video camera, a fish-eye lens, etc., then a frame of environmental scene image is selected from the environmental scene image acquired in real time as a1 st frame of environmental scene image, then an image registration is performed on adjacent frames of environmental scene images by using a Simultaneous Localization and Mapping (Slam) algorithm, and three-dimensional point cloud data of the 1 st frame of environmental scene image and each frame of environmental scene image thereafter are calculated, where the three-dimensional point cloud data includes three-dimensional coordinates of each feature point. Of course, the three-dimensional point cloud data further includes color information of each feature point, and the application is not particularly limited.
In this embodiment of the application, the 1 st frame of environmental scene image is an initial frame of image, and it is known that the virtual information is first superimposed on the 1 st frame of environmental scene image.
Specifically, when the 1 st frame of environmental scene image and the three-dimensional point cloud data of each frame of environmental scene image behind the 1 st frame of environmental scene image are calculated through the Slam algorithm, the Slam algorithm is used for carrying out image registration on an environmental scene image containing the three-dimensional point cloud data to be calculated and one or more frames of environmental scene images adjacent to the environmental scene image, and therefore the three-dimensional point cloud data of the frame of environmental scene image are calculated; for example, the Slam algorithm may be used to perform image registration on the 1 st frame of environmental scene image and one or more previous frames of environmental scene images, so as to calculate three-dimensional point cloud data of the 1 st frame of environmental scene image.
Step S102 is executed next, and steps S103 to S105 are executed for the 1 st frame environmental scene image and each frame environmental scene image thereafter in sequence.
In step S103, the three-dimensional point cloud data of the i-th frame environmental scene image calculated in step S102 is compared with the three-dimensional point cloud data of the (i-1) th frame environmental scene image, and the three-dimensional point cloud data not included in the three-dimensional point cloud data of the (i-1) th frame environmental scene image is found from the three-dimensional point cloud data of the i-th frame environmental scene image as the newly added three-dimensional point cloud data. And after the newly-added three-dimensional point cloud data is obtained, processing the newly-added three-dimensional point cloud data by utilizing a Delaunay triangulation algorithm to obtain a newly-added triangular surface set.
The Delaunay triangulation algorithm is introduced here: the Delaunay triangulation algorithm is used for a network formed by Delaunay triangulation of a set of points, and two important criteria must be met to satisfy the definition of Delaunay triangulation:
1) empty circle characteristic: the Delaunay triangulation network is unique (any four points cannot be in a circle), and no other point exists in the range of the circumscribed circle of any triangle in the Delaunay triangulation network;
2) maximizing the minimum angular characteristic: in the triangulation possibly formed by the scatter set, the minimum angle of the triangle formed by the Delaunay triangulation is the largest. In this sense, the Delaunay triangulation network is the "nearest to regularized" triangulation network. Specifically, the minimum angle of six internal angles is not increased after two adjacent triangles form the diagonal of the convex quadrangle and are mutually exchanged.
The Delaunay triangulation algorithm has the following excellent characteristics:
1) the closest is: forming a triangle by the nearest three points, wherein all line segments (sides of the triangle) are not intersected;
2) uniqueness: consistent results will be obtained no matter where the region is constructed;
3) optimality: if the diagonals of the convex quadrangle formed by any two adjacent triangles can be interchanged, the smallest angle in six interior angles of the two triangles cannot be enlarged;
4) most regular: if the minimum angles of each triangle in the triangulation network are arranged in an ascending order, the numerical value obtained by the arrangement of the Delaunay triangulation network is the maximum;
5) regionality: only the adjacent triangle is influenced when a certain vertex is added, deleted or moved;
6) housing with convex polygon: the outermost boundaries of the triangulation network form a convex polygonal outer shell.
Therefore, the accuracy of the newly added triangular surface set obtained by processing the newly added three-dimensional point cloud data by using the Delaunay triangulation algorithm is greatly improved, and the accuracy of a result obtained by calculating by using the newly added triangular surface set in the later step is also improved.
Specifically, when i is equal to 1, new three-dimensional point cloud data of the 1 st frame relative to the 0 th frame environmental scene image needs to be acquired, and the new three-dimensional point cloud data is processed by using a Delaunay triangulation algorithm to obtain a processed new triangular surface set.
In the embodiment of the application, the 1 st frame environmental scene image is an initial frame image, so that the three-dimensional point cloud data of the 0 th frame environmental scene image is 0, and therefore the three-dimensional point cloud data of the 1 st frame environmental scene image can be used as new three-dimensional point cloud data of the 1 st frame environmental scene image; and then, processing the three-dimensional point cloud data of the 1 st frame of environmental scene image by utilizing a Delaunay triangulation algorithm to obtain a newly-added triangular surface set of the 1 st frame of environmental scene image formed by all the processed triangular surfaces.
For example, an image a in an environment scene image acquired in real time by a camera device is a1 st frame environment image, three-dimensional point cloud data d of the image a comprises c, c1, c2, b11 and b12, and since the image a is the 1 st frame environment scene image, newly-added three-dimensional point cloud data corresponding to the image a can be determined to be d, and then c, c1, c2, b11 and b12 are processed by using a Delaunay triangulation algorithm, so that a newly-added triangular surface set of the image a is formed by all processed triangular surfaces; and when the a1 image in the acquired environmental scene image is the 2 nd frame environmental scene image, comparing the three-dimensional point cloud data d1 of the a1 image with the three-dimensional point cloud data d of the a image, if the three-dimensional point cloud data contained in d1 are c2, b11, b12, b13, b14, b15, c8 and c9, comparing to obtain newly-added three-dimensional point cloud data as b13, b14, b15, c8 and c9, and then processing b13, b14, b15, c8 and c9 by using a Delaunay triangulation algorithm to obtain a newly-added triangular surface set of the a1 image formed by all the processed triangular surfaces.
And step S104 is executed, each newly added triangular surface in the newly added triangular surface set is spliced into the plane set of the (i-1) th frame environment scene image to obtain the plane set of the i th frame environment scene image, wherein the plane set comprises the vertex set and the normal direction of each plane.
In a specific implementation process, when i is 1, because the 1 st frame environmental scene image is an initial frame image, and the plane set of the (i-1) th frame environmental scene image is 0, triangular plane stitching may be performed on a certain three-dimensional point in the 1 st frame environmental scene image or a certain newly added triangular plane in the newly added triangular plane set, and all planes obtained by the stitching may constitute the plane set of the 1 st frame environmental scene image.
Specifically, n is sequentially taken from 1 to i, so that from the 1 st frame of environmental scene image to the nth frame of environmental scene image, each newly added triangular surface in a newly added triangular surface set of a next frame relative to a previous frame is spliced into a plane set of the previous frame of environmental scene image, that is, a plane set of the next frame of environmental scene image can be obtained, and further, only splicing operation needs to be performed on newly added three-dimensional point cloud data, that is, the plane set of the next frame of environmental scene image can be obtained; the three-dimensional point cloud data of each frame of environment scene image does not need to be calculated to obtain a plane set, so that the data volume required to be carried out when the plane set of each frame of environment scene image is obtained is greatly reduced, the larger the n value is, the higher the data volume reduction degree is, and on the basis of improving the accuracy of the newly added triangular surface set, the accuracy of the plane set obtained by splicing each newly added triangular surface is improved; and on the basis of greatly reducing the calculated amount, the real-time performance of acquiring the plane set of the i-th frame of environmental scene image is greatly improved, and the real-time performance of the virtual information superposition display is improved accordingly.
For example, the a image is the initial plane set corresponding to the 1 st frame environmental scene image and is b, and when the current frame environmental scene image is changed from the a image to the a1 image, if the newly added triangular plane set of the a1 image relative to the a image is represented by b1, the plane set of the a1 image can be determined to be (b + b 1); if the sequence of the current environmental scene image updating is a2, a3 and a4 … an in turn, and the corresponding newly added triangular surface sets are represented by b2, b3 and b4 … bn in turn, the plane sets of a2, a3 and a4 … an are determined to be (b + b1+ b2), (b + b1+ b2+ b3) and (b + b1+ b2+ b3+ b4) … (b + b1+ b2+ b3+ b4+ … bn) in turn.
In the embodiment of the application, each newly added triangular surface can be spliced into the plane set of the (i-1) th frame environmental scene image by using the plane in the plane set of the (i-1) th frame environmental scene image and the normal relation of each newly added triangular surface in the newly added triangular surface set, and all planes obtained by splicing form the i-th frame environmental scene image.
The normal direction of each newly added triangular surface is calculated according to three-dimensional coordinates of a plurality of three-dimensional points contained in the newly added triangular surface; similarly, the normal direction of each plane is also calculated according to the three-dimensional coordinates of a plurality of three-dimensional points contained in the plane; and (3) comparing the normal direction of the plane with the normal direction of the newly added triangular surface to obtain the normal relation between the plane in the plane set of the (i-1) th frame environmental scene image and each newly added triangular surface in the newly added triangular surface set.
In this embodiment of the application, the step of splicing each triangular surface in the newly added triangular surface set to the plane set of the (i-1) th frame environmental scene image to obtain the plane set of the i th frame environmental scene image may be implemented by the following steps, which specifically include:
step 1: determining all planes adjacent to the newly added triangular surface in the newly added triangular surface set from the plane set of the (i-1) th frame environmental scene image to form a boundary plane according to the newly added triangular surface data;
step 2: then expanding the boundary plane towards the direction of the adjacent newly-added triangular surface, and calculating the normal relation between the adjacent newly-added triangular surface and the boundary plane;
and step 3: judging whether the adjacent newly-added triangular surfaces and the boundary plane are coplanar or not by utilizing the calculated normal relation; if the newly increased triangular surfaces are coplanar, splicing the adjacent newly increased triangular surfaces into the boundary plane, and updating the boundary information of the boundary plane; if not, taking the adjacent newly-added triangular surface as a new plane, and adding the adjacent newly-added triangular surface into the boundary plane, wherein the boundary information of each plane is the outermost vertex set and the normal direction in the triangular surface set forming the plane;
and (5) repeatedly executing the step (1) to the step (3) until all the newly added triangular surfaces in the newly added triangular surface set are calculated, and taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image.
The boundary information of each plane is a set of vertices and a normal direction of an outermost periphery in a set of triangular surfaces forming the plane, and may also include a center coordinate of the plane and area information of a maximum inscribed rectangle.
For example, when the current frame environment scene image is changed from an image a to an image a1, each newly added triangular surface in b1 is spliced into b, then, according to the triangular surface data of b1, the boundary planes of the b1 vector are determined to be c, c1 and c2 from b, if the newly added triangular surfaces in b1 are b11, b12, b13, b14 and b15, then, according to the boundary information of c, c1 and c2, then, the triangular surfaces adjacent to c, c1 and c2 are determined to be b11, b12 and b13 from b 1; then c, c1 and c2 are expanded towards the directions of b11, b12 and b13, and the normal relations between each newly added triangular surface in b11, b12 and b13 and c, c1 and c2 are respectively calculated; if the normal directions of c and b11 are the same, the normal directions of c1 and b12 are the same, and the normal directions of c2 and b13 are different, then c and b11 are coplanar, c1 and b12 are coplanar, and c2 and b13 are not coplanar; thus, b11 is spliced into the slave c, and the boundary information of c is updated; and stitching c1 into b12 and updating the boundary information of c 1; and b13 as a new plane and b13 is added to the bounding plane.
At this time, the boundary planes are changed to c, c1, c2, and b 13; then c, c1, c2 and b13 are expanded towards the directions of b14 and b15, if the normal directions of b14 and b13 are determined to be the same according to the calculated normal relation, and the normal directions of b15 and c2 are determined to be the same, b14 is spliced into b13, and the boundary information of b13 is updated; and stitching b15 into c2 and updating the boundary information of c 2; thus, a set of planes (b + b1) of the a1 image is obtained, in which (b + b1) the boundary information of c, c1 and c2 has changed compared to b, and a plane b13 is added.
In another embodiment of the present application, the taking the calculated plane set as the plane set of the ith frame of environmental scene image specifically includes, as shown in fig. 2:
s201, detecting whether the area of each plane in the plane set obtained through calculation is not smaller than a first preset area;
s202, when a plane smaller than the first preset area is detected, according to the normal relation between the plane smaller than the first preset area and the adjacent plane, splicing the plane smaller than the first preset area to the adjacent plane which is most matched with the normal of the plane, until all planes smaller than the first preset area in the calculated plane set are spliced to the adjacent planes, and obtaining the plane set of the ith frame of environmental scene image.
When the plane smaller than the first preset area is spliced to the adjacent plane which is most matched with the normal line of the plane, the boundary information of the most matched adjacent plane is updated.
Specifically, the first preset area may be determined by the calculation performance of hardware, and the higher the calculation performance of hardware, the smaller the first preset area is set; on the contrary, the worse the calculation performance of the hardware is, the larger the first preset area is set; therefore, the size of the first preset area is set through the performance of hardware, the number of planes in a plane set used for each frame of environment scene image can be effectively controlled, the calculated amount of splicing the newly added triangular surface to the plane set is reduced, the calculation time is shortened, and the real-time property of virtual information superposed to a target area can be ensured.
For example, when the current frame environment scene image is changed from an a image to an a1 image, each newly added triangular surface in b1 is spliced into b, so that a plane set (b + b1) of a1 image is obtained, wherein (b + b1) compared with b, the boundary information of c, c1 and c2 is changed, and a plane b13 is newly added; judging whether the area of each plane in (b + b1) is not less than the first preset area, wherein since each plane in b has been judged once, it is known that the area of each plane in b is necessarily greater than the first preset area, and therefore, only the area of the newly added plane b13 needs to be judged; if the area of b13 is not less than the first preset area, determining that the plane in (b + b1) is one more plane than b, b13 and the boundary information of c, c1 and c2 has changed; if the area of b13 is smaller than the first preset area, acquiring adjacent planes of b13 as c1, c2 and b 11; if the normal direction of b13 is calculated to be the closest match to the normal direction of c2 through the normal, b13 is spliced into c2, and the boundary information of c2 is updated again, so that it can be determined that the number of the resulting planes in (b + b1) is the same as that of the planes compared with b, but the boundary information of c, c1 and c2 is changed.
In another embodiment of the present application, after obtaining the plane set of the ith frame of environmental scene image, the method further includes: and marking each plane in the plane set of the ith frame of environmental scene image in an index mode so as to quickly search and identify the corresponding plane through the index mark, thereby improving the searching efficiency.
Specifically, after each plane in the plane set of the ith frame of environment scene image is indexed, acquiring an index mark set corresponding to the plane set of the ith frame of environment scene image; then, the area of each plane in the plane set corresponding to the index mark set can be sorted, and each plane can be sorted according to the size of the area; or sorting each plane from small to large in area to obtain area sorting of planes in the plane set of the ith frame of environmental scene image.
Specifically, the area of each plane in the plane set of the ith frame of environmental scene image may be calculated according to the boundary information of each plane; and then, by using the calculated plane of each plane, sorting the area of each plane through the index mark set, thereby obtaining the area sorting of the planes in the plane set of the ith frame of environmental scene image.
For example, the initial plane set corresponding to the 1 st frame environmental scene image as the a image includes c, c1 and c2 images in b, c may be marked with 001, c1 with 002, and c2 with 003, when the current frame environmental scene image changes from the a image to the a1 image, the boundary information of c, c1 and c2 in (b + b1) is updated, b13 is marked with 006, and the boundary information of b13 is recorded; and each plane in (b + b1) can be sorted in turn according to the area size of the plane, and the area sorting of the plane in (b + b1) can be obtained, for example, c2> c > c1> b13, that is, the area of c2 is the largest, and then c, c1 and b13 are carried out in turn.
In another embodiment of the present application, after the plane set of the ith frame of environmental scene image obtained through step S104, the method further includes:
detecting whether the plane set of the ith frame environment scene image is updated compared with the plane set of the (i-1) th frame environment scene image; if not, processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm, and forming a triangular surface set of the ith frame of environmental scene image by using all triangular surfaces obtained by processing; performing triangular surface splicing by taking a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and forming a plane set of the ith frame of environmental scene image by all planes obtained by splicing; if updated, the process continues to step S105.
In a specific implementation process, whether the plane set of the ith frame and the (i-1) th frame environment scene image is updated or not may be detected by detecting whether boundary information of planes in the plane set of the ith frame environment scene image and the (i-1) th frame environment scene image changes or not, and/or whether the number of planes in the plane set of the ith frame environment scene image and the (i-1) th frame environment scene image changes or not.
Specifically, if it is detected that the boundary information and the number of planes in the plane set of the i-th frame environmental scene image are changed from those in the (i-1) th frame, it may be determined that the plane set of the i-th frame environmental scene image is updated from those in the (i-1) th frame; if it is detected that neither the boundary information nor the number of planes in the plane set of the ith frame and the (i-1) th frame environmental scene image has changed, it may be determined that the plane set of the ith frame and the (i-1) th frame environmental scene image has not been updated.
For example, when the current frame environment scene image is changed from an a image to an a1 image, each newly added triangular surface in b1 is spliced into b, so that a plane set (b + b1) of an a1 image is obtained, wherein (b + b1) compared with b, the boundary information of c, c1 and c2 is changed, and a plane b13 is newly added, so that it can be determined that the plane set of a1 compared with a is changed.
Specifically, in the process of forming the stitched planes into the plane set of the ith frame of environmental scene image, each plane in the plane set of the ith frame of environmental scene image may be marked with an index, and then the index and the boundary information of each plane in the plane set of the ith frame of environmental scene image may be recorded.
Specifically, whether the ith frame and the (i-1) th frame environmental scene image are changed or not can be determined by whether the plane set of the ith frame environmental scene image is updated compared with the plane set of the (i-1) th frame environmental scene image or not; in the practical application process, because each two adjacent frames of environmental scene images generally change, when it is detected whether the plane set of the ith frame of environmental scene image is updated compared with the plane set of the (i-1) th frame of environmental scene image, there is a great probability that it can be determined that the plane set of the ith frame of environmental scene image is obtained incorrectly; at this time, the planar set of the ith frame of environmental scene image is acquired by using the three-dimensional point cloud data of the ith frame of environmental scene image, so that the accuracy of the acquired planar set of the ith frame of environmental scene image can be effectively improved.
Step S105 is executed next, based on a preset condition, a visible plane is selected from the plane set of the ith frame of environmental scene image as a target plane, and the virtual information to be displayed is superimposed on the selected target plane for display.
In the specific implementation process, the watching area of the user can be collected in real time; when the fact that the duration of a first watching area of a user reaches preset time is detected, at least one visible plane located in the first watching area is obtained from the plane set of the ith frame of environment scene image, then one visible plane is selected from the at least one visible plane to serve as the target plane, and the virtual information is overlapped in the target plane to be displayed.
Specifically, the rotation of the eyes of the user can be acquired in real time through a camera, so that the gazing area of the user can be acquired from the ith frame of environmental scene image in real time; after the watching areas of the user are collected in real time, acquiring the duration of each watching area of the user, and detecting whether the duration of each watching area of the user reaches the preset time; if the fact that the duration of a certain watching area of the user reaches the preset time is detected, taking the watching area as the first watching area; and then selecting at least one visible plane in the first gazing area, selecting one visible plane from the at least one visible plane as the target plane according to the principles of area priority, center priority and the like, and overlapping the virtual information in the selected visible plane for display.
In this embodiment, the area-first principle refers to selecting a visible plane with the largest area from the at least one visible plane as the target plane, and the center-first principle refers to selecting a visible plane closest to the center of the first gaze area from the at least one visible plane as the target plane.
In the embodiment of the present application, the preset time is set according to an actual situation, and may also be set by a device or a user, where the preset time is generally set to a value not less than 2 seconds(s), for example, 2s, 3s, 5s, and the like.
In this embodiment of the application, the visible plane refers to a plane that can be seen by a user in the ith frame of environment scene image, and whether each plane is visible or not can be calculated through a normal line of each plane in the plane set of the ith frame of environment scene image, and if it is calculated that a certain plane is visible, the plane is determined to be a visible plane; and if the certain plane is calculated to be invisible, judging the plane to be a non-visible plane. Of course, in order to improve the accuracy of the determination, it may also be calculated whether each plane is a visible plane by the center coordinate and the normal of each plane in the plane set of the ith frame of environmental scene image.
In a specific implementation process, a throwing track for throwing the virtual information may be further acquired, a visible plane matched with the throwing track is acquired from the plane set of the ith frame of environment scene image as the target plane, and the virtual information is superimposed on the target plane for display.
Specifically, a throwing gesture for directing the virtual information to a target area may be acquired when the virtual information is requested; and calculating the throwing track according to the throwing gesture, calculating a visible plane closest to the throwing track by using the throwing track as the target plane, and finally superposing the virtual information in the target plane for displaying.
Specifically, when the throwing track is used to calculate a visible plane closest to the throwing track as the target plane, it may be further determined whether an area of the closest visible plane is not less than an occupied area for displaying the virtual information; if the area of the nearest visible plane is judged to be not smaller than the occupied area, the nearest visible plane is used as the target plane, and the virtual information is superposed in the target plane to be displayed; and if the area of the nearest visible plane is judged to be smaller than the occupied area, selecting one visible plane larger than the occupied area from the plane set of the ith frame of environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Specifically, when it is determined that the area of the nearest visible plane is smaller than the occupied area, any one visible plane larger than the occupied area may be selected from the plane set of the ith frame of environmental scene image as the target plane, or a visible plane closest to the throwing track may be selected from all visible planes larger than the occupied area as the target plane.
For example, when the virtual information is a television news interface, the occupied area of the television news interface is 32 centimeters (mm) by 18mm, and the area of the visible plane closest to the throwing track can be calculated to be 35mm by 25mm by using the throwing track, and since 35mm by 25mm >32mm by 18mm, the nearest visible plane can be determined to be the target plane; if the nearest visible plane is 25mm × 12mm, because 25mm × 12mm <32mm × 18mm, according to the area ranking of the planes in the plane set of the i-th frame environmental scene image, any one visible plane larger than 32mm × 18mm is selected from the plane set of the i-th frame environmental scene image as the target plane, and one visible plane nearest to the throwing track may also be selected from all visible planes larger than 32mm × 18mm as the target plane.
In this embodiment of the application, after the virtual information is superimposed on the target plane and displayed, the method further includes: acquiring gesture operation for moving the virtual information; according to the gesture operation, the virtual information is moved and superposed on the visible plane where the gesture operation finally stops for displaying.
In another embodiment of the present application, after obtaining the plane set of the ith frame of environmental scene image through step S104, the method further includes:
detecting whether the sum of the areas of newly added triangular surfaces in each plane in the plane set of the ith frame of environmental scene image is not smaller than a second preset area; when the sum of the areas of newly added triangular surfaces in a certain plane is detected to be smaller than the second preset area, processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a triangular surface set of the ith frame of environmental scene image; then, carrying out triangular surface splicing by taking a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and forming a new plane set of the ith frame of environmental scene image by all planes obtained by splicing; and then selecting a visible plane from the new plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and overlapping the virtual information to be displayed in the selected target plane for displaying.
Certainly, it may also be detected whether a newly-added triangular surface is continuous in each plane in the plane set of the i-th frame of environmental scene image, and if not, the three-dimensional point cloud data of the i-th frame of environmental scene image is processed by using a Delaunay triangulation algorithm to obtain a triangular surface set of the i-th frame of environmental scene image; then, carrying out triangular surface splicing by taking a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and forming a new plane set of the ith frame of environmental scene image by all planes obtained by splicing; and then selecting a visible plane from the new plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and overlapping the virtual information to be displayed in the selected target plane for displaying.
Based on the same technical concept as the above method, the present invention further provides an apparatus for implementing augmented reality, as shown in fig. 3, including:
a three-dimensional point cloud data obtaining unit 301, configured to select a certain frame of environmental scene image from environmental scene images acquired in real time as a1 st frame of environmental scene image, perform image registration on adjacent multiple frames of environmental scene images by using a Slam algorithm, and calculate three-dimensional point cloud data of the 1 st frame of environmental scene image and each frame of environmental scene image thereafter;
a loop execution unit 302, configured to execute the following steps for the 1 st frame of environmental scene image and each frame of environmental scene image after the 1 st frame of environmental scene image in sequence: acquiring newly-added three-dimensional point cloud data of an ith frame relative to an (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set, wherein i is sequentially taken from 1 to n, and n is an integer not less than 2; splicing each newly added triangular surface in the newly added triangular surface set into the plane set of the (i-1) th frame environment scene image to obtain a plane set of the i th frame environment scene image, wherein the plane set comprises a vertex set and a normal direction of each plane; and selecting a visible plane from the plane set of the ith frame of environment scene image as a target plane based on a preset condition, and superposing virtual information to be displayed on the selected target plane for displaying, wherein whether the plane is the visible plane is judged based on the normal direction of each plane.
Preferably, the loop execution unit 302 further includes:
and the newly added three-dimensional point cloud data acquisition unit is used for comparing the calculated three-dimensional point cloud data of the ith frame of environmental scene image with the three-dimensional point cloud data of the (i-1) th frame of environmental scene image to obtain the newly added three-dimensional point cloud data.
Preferably, the loop execution unit 302 further includes:
a newly-added triangular surface set acquisition unit, configured to use the three-dimensional point cloud data of the 1 st frame of environmental scene image as newly-added three-dimensional point cloud data of the 1 st frame of environmental scene image when i is equal to 1; and processing the three-dimensional point cloud data of the 1 st frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a newly-added triangular surface set of the 1 st frame of environmental scene image formed by all the processed triangular surfaces.
Preferably, the loop execution unit 302 further includes:
and the plane set acquisition unit is used for performing triangular plane splicing on the newly added triangular plane set of the 1 st frame environmental scene image by taking a certain three-dimensional point in the 1 st frame environmental scene image as a reference when i is equal to 1, and forming the plane set of the 1 st frame environmental scene image by all planes obtained through splicing.
Preferably, the plane set obtaining unit is further configured to determine, from the plane set of the ith frame of environmental scene image, all planes adjacent to the newly added triangular surface in the newly added triangular surface set to form a boundary plane; expanding the boundary plane towards the direction of the adjacent newly-added triangular surface, and calculating the normal relation between the adjacent newly-added triangular surface and the boundary plane; judging whether the adjacent newly-added triangular surfaces and the boundary plane are coplanar or not by utilizing the calculated normal relation; if the newly increased triangular surfaces are coplanar, splicing the adjacent newly increased triangular surfaces into the boundary plane, and updating the boundary information of the boundary plane; if not, taking the adjacent newly-added triangular surface as a new plane, and adding the adjacent newly-added triangular surface into the boundary plane, wherein the boundary information of each plane is the outermost vertex set and the normal direction in the triangular surface set forming the plane; and repeatedly executing the steps until all the newly-added triangular surfaces in the newly-added triangular surface set are calculated, and taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image.
Preferably, the plane set obtaining unit is further configured to detect whether an area of each plane in the calculated plane set is not smaller than a first preset area; and when a plane smaller than the first preset area is detected, splicing the plane smaller than the first preset area into an adjacent plane which is most matched with the normal of the plane according to the normal relation until all planes smaller than the first preset area in the calculated plane set are spliced into the adjacent plane, so as to obtain the plane set of the ith frame of environmental scene image.
Preferably, the loop execution unit 302 further includes:
the superposition display unit is used for acquiring a throwing track for throwing the virtual information; and acquiring a visible plane closest to the throwing track from the plane set of the ith frame environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Preferably, the apparatus further comprises:
an area determination unit, configured to determine whether an area of a nearest visible plane closest to the throwing trajectory is not smaller than an occupied area for displaying the virtual information when the visible plane closest to the throwing trajectory is acquired from the plane set of the ith frame of environmental scene image as the target plane;
the superposition display unit is used for taking the nearest visible plane as the target plane and superposing the virtual information in the target plane for display when the area of the nearest visible plane is judged to be not smaller than the occupied area; and when the area of the nearest visible plane is judged to be smaller than the occupied area, selecting one visible plane larger than the occupied area from the plane set of the ith frame of environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
Preferably, the apparatus further comprises:
a plane set update detection unit, configured to detect whether the plane set of the ith frame environmental scene image is updated compared with the plane set of the (i-1) th frame environmental scene image after obtaining the plane set of the ith frame environmental scene image;
the triangular surface set acquisition unit is used for processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm when the situation that the image is not updated is detected, and forming a triangular surface set of the ith frame of environmental scene image by using all triangular surfaces obtained by processing;
the plane set obtaining unit is further configured to perform triangular surface stitching with a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and form a plane set of the ith frame of environmental scene image with all planes obtained through stitching.
Preferably, the apparatus further comprises:
and the index marking unit is used for marking each plane in the plane set of the ith frame environment scene image in an index mode after the plane set of the ith frame environment scene image is obtained.
Preferably, the apparatus further comprises:
the triangular surface area detection unit is used for detecting whether the sum of the areas of newly added triangular surfaces in each plane in the plane set of the ith frame of environment scene image is not less than a second preset area after the plane set of the ith frame of environment scene image is obtained;
the triangular surface set obtaining unit is further configured to, when it is detected that the sum of the areas of newly added triangular surfaces in a certain plane is smaller than the second preset area, process the three-dimensional point cloud data of the i-th frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a triangular surface set of the i-th frame of environmental scene image;
the plane set acquisition unit is further configured to perform triangular surface stitching with a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and form a new plane set of the ith frame of environmental scene image with all planes obtained by stitching;
and the superposition display unit is also used for selecting a visible plane from the new plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and superposing the virtual information to be displayed on the selected target plane for displaying.
The invention has the following beneficial effects:
the method comprises the steps of processing newly-added three-dimensional point cloud data of two adjacent frames of environmental scene images to obtain a plane set of each frame of environmental scene image; then based on preset conditions, selecting a visible plane from the plane set of each frame of the environment scene image as a target plane, and the virtual information to be displayed is superposed in the selected target plane for display, so that the embodiment of the application automatically calculates the target plane through equipment, then the virtual object is overlaid in the target plane for displaying, without the user manually operating the virtual object to move the virtual object to the target plane for displaying, thereby realizing the automatic control of the virtual object to move to the position to be displayed, and the accuracy of the target plane calculated by the device is obviously improved compared with the accuracy of manual movement, and need not user manual operation, so, realized simplifying operation and can effectively reduce the effect of the deviation that virtual reality fuses to exist.
The modules or units in the embodiments of the present invention may be implemented by a general-purpose integrated circuit, such as a CPU (central processing Unit), or an ASIC (Application Specific integrated circuit).
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims (13)

1. A method for implementing augmented reality, comprising:
selecting a certain frame of environmental scene image from the environmental scene images collected in real time as a1 st frame of environmental scene image, performing image registration on adjacent frames of environmental scene images by using a Slam algorithm, and calculating the 1 st frame of environmental scene image and three-dimensional point cloud data of each frame of environmental scene image;
sequentially executing the following steps for the 1 st frame environmental scene image and each frame environmental scene image after the 1 st frame environmental scene image:
acquiring newly-added three-dimensional point cloud data of an ith frame relative to an (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set, wherein i is sequentially taken from 1 to n, and n is an integer not less than 2;
splicing each newly added triangular surface in the newly added triangular surface set into the plane set of the (i-1) th frame environment scene image to obtain a plane set of the i th frame environment scene image, wherein the plane set comprises a vertex set and a normal direction of each plane;
and selecting a visible plane from the plane set of the ith frame of environment scene image as a target plane based on a preset condition, and superposing virtual information to be displayed on the selected target plane for displaying, wherein whether the plane is the visible plane is judged based on the normal direction of each plane.
2. The method of claim 1, wherein the acquiring of the newly added three-dimensional point cloud data of the ith frame relative to the (i-1) th frame environmental scene image specifically comprises:
and comparing the calculated three-dimensional point cloud data of the ith frame of environmental scene image with the three-dimensional point cloud data of the (i-1) th frame of environmental scene image to obtain the newly-added three-dimensional point cloud data.
3. The method according to claim 1, wherein when i is 1, the acquiring new three-dimensional point cloud data of the ith frame relative to the (i-1) th frame environmental scene image, and processing the new three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed new triangular surface set specifically includes:
taking the three-dimensional point cloud data of the 1 st frame of environmental scene image as newly-added three-dimensional point cloud data of the 1 st frame of environmental scene image;
and processing the three-dimensional point cloud data of the 1 st frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a newly-added triangular surface set of the 1 st frame of environmental scene image formed by all the processed triangular surfaces.
4. The method as claimed in claim 3, wherein when i is 1, the stitching each new triangle plane in the set of new triangle planes into the set of planes of the (i-1) th frame environmental scene image to obtain the set of planes of the i-th frame environmental scene image includes:
and performing triangular surface splicing on the newly added triangular surface set of the 1 st frame environmental scene image by taking a certain three-dimensional point in the 1 st frame environmental scene image as a reference, and forming a plane set of the 1 st frame environmental scene image by all planes obtained by splicing.
5. The method as claimed in claim 1, wherein the stitching each of the newly added triangular faces in the newly added triangular face set to the plane set of the (i-1) th frame environmental scene image to obtain the plane set of the i-th frame environmental scene image includes:
step 1: determining all planes adjacent to the newly added triangular surface in the newly added triangular surface set from the plane set of the ith frame of environmental scene image to form a boundary plane;
step 2: expanding the boundary plane towards the direction of the adjacent newly-added triangular surface, and calculating the normal relation between the adjacent newly-added triangular surface and the boundary plane;
and step 3: judging whether the adjacent newly-added triangular surfaces and the boundary plane are coplanar or not by utilizing the calculated normal relation; if the newly increased triangular surfaces are coplanar, splicing the adjacent newly increased triangular surfaces into the boundary plane, and updating the boundary information of the boundary plane; if not, taking the adjacent newly-added triangular surface as a new plane, and adding the adjacent newly-added triangular surface into the boundary plane, wherein the boundary information of each plane is the outermost vertex set and the normal direction in the triangular surface set forming the plane;
and (5) repeatedly executing the step (1) to the step (3) until all the newly added triangular surfaces in the newly added triangular surface set are calculated, and taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image.
6. The method according to claim 5, wherein the using the calculated plane set as the plane set of the ith frame of environmental scene image specifically includes:
detecting whether the area of each plane in the calculated plane set is not smaller than a first preset area or not;
and when a plane smaller than the first preset area is detected, splicing the plane smaller than the first preset area into an adjacent plane which is most matched with the normal of the plane according to the normal relation until all planes smaller than the first preset area in the calculated plane set are spliced into the adjacent plane, so as to obtain the plane set of the ith frame of environmental scene image.
7. The method according to claim 1, wherein based on a preset condition, selecting a visible plane from the plane set of the ith frame of environmental scene image as a target plane, and superimposing virtual information to be displayed on the selected target plane for display, specifically:
collecting a throwing track for throwing the virtual information;
and acquiring a visible plane closest to the throwing track from the plane set of the ith frame environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
8. The method of claim 7, wherein when a visible plane closest to the throwing trajectory is acquired as the target plane from the set of planes of the ith frame environmental scene image, the method further comprises:
judging whether the area of the nearest visible plane is not smaller than the occupied area for displaying the virtual information;
if the area of the nearest visible plane is judged to be not smaller than the occupied area, the nearest visible plane is used as the target plane, and the virtual information is superposed in the target plane to be displayed;
and if the area of the nearest visible plane is judged to be smaller than the occupied area, selecting one visible plane larger than the occupied area from the plane set of the ith frame of environment scene image as the target plane, and overlapping the virtual information in the target plane for displaying.
9. The method of claim 1, wherein after obtaining the set of planes for the ith frame of image of the environmental scene, the method further comprises:
detecting whether the plane set of the ith frame environment scene image is updated compared with the plane set of the (i-1) th frame environment scene image;
if not, processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm, and forming a triangular surface set of the ith frame of environmental scene image by using all triangular surfaces obtained by processing;
and performing triangular surface splicing by taking a certain triangular surface in the triangular surface set of the ith frame of environmental scene image as a reference, and forming a plane set of the ith frame of environmental scene image by all planes obtained by splicing.
10. The method of claim 1, wherein after obtaining the set of planes for the ith frame of image of the environmental scene, the method further comprises:
and marking each plane in the plane set of the ith frame of environmental scene image in an indexing mode.
11. The method of claim 1, wherein after obtaining the set of planes for the ith frame of image of the environmental scene, the method further comprises:
detecting whether the sum of the areas of newly added triangular surfaces in each plane in the plane set of the ith frame of environmental scene image is not smaller than a second preset area;
when the sum of the areas of newly added triangular surfaces in a certain plane is detected to be smaller than the second preset area, processing the three-dimensional point cloud data of the ith frame of environmental scene image by using a Delaunay triangulation algorithm to obtain a triangular surface set of the ith frame of environmental scene image;
performing triangle surface splicing by taking a certain triangle surface in the triangle surface set of the ith frame of environment scene image as a reference, and forming a new plane set of the ith frame of environment scene image by all planes obtained by splicing;
and selecting a visible plane from the new plane set of the ith frame of environmental scene image as a target plane based on a preset condition, and overlapping the virtual information to be displayed in the selected target plane for displaying.
12. An apparatus for implementing augmented reality, comprising:
the system comprises a three-dimensional point cloud data acquisition unit, a processing unit and a processing unit, wherein the three-dimensional point cloud data acquisition unit is used for selecting a certain frame of environmental scene image from environmental scene images acquired in real time as a1 st frame of environmental scene image, carrying out image registration on adjacent frames of environmental scene images by using a Slam algorithm, and calculating the 1 st frame of environmental scene image and three-dimensional point cloud data of each frame of environmental scene image behind the 1 st frame of environmental scene image;
a loop execution unit, configured to sequentially execute the following steps for the 1 st frame of environmental scene image and each frame of environmental scene image thereafter: acquiring newly-added three-dimensional point cloud data of an ith frame relative to an (i-1) th frame environmental scene image, and processing the newly-added three-dimensional point cloud data by using a Delaunay triangulation algorithm to obtain a processed newly-added triangular surface set, wherein i is sequentially taken from 1 to n, and n is an integer not less than 2; splicing each newly added triangular surface in the newly added triangular surface set into the plane set of the (i-1) th frame environment scene image to obtain a plane set of the i th frame environment scene image, wherein the plane set comprises a vertex set and a normal direction of each plane; and selecting a visible plane from the plane set of the ith frame of environment scene image as a target plane based on a preset condition, and superposing virtual information to be displayed on the selected target plane for displaying, wherein whether the plane is the visible plane is judged based on the normal direction of each plane.
13. The apparatus of claim 12, wherein the loop execution unit further comprises:
a plane set acquisition unit, configured to determine, from the plane set of the ith frame of environmental scene image, a boundary plane formed by all planes adjacent to the newly added triangular surface in the newly added triangular surface set; expanding the boundary plane towards the direction of the adjacent newly-added triangular surface, and calculating the normal relation between the adjacent newly-added triangular surface and the boundary plane; judging whether the adjacent newly-added triangular surfaces and the boundary plane are coplanar or not by utilizing the calculated normal relation; if the newly increased triangular surfaces are coplanar, splicing the adjacent newly increased triangular surfaces into the boundary plane, and updating the boundary information of the boundary plane; if not, taking the adjacent newly-added triangular surface as a new plane, and adding the adjacent newly-added triangular surface into the boundary plane, wherein the boundary information of each plane is the outermost vertex set and the normal direction in the triangular surface set forming the plane; and repeatedly executing the steps until all the newly-added triangular surfaces in the newly-added triangular surface set are calculated, and taking the plane set obtained by calculation as the plane set of the ith frame of environmental scene image.
CN201710181831.7A 2017-03-24 2017-03-24 Method and equipment for realizing augmented reality Active CN108629799B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710181831.7A CN108629799B (en) 2017-03-24 2017-03-24 Method and equipment for realizing augmented reality

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710181831.7A CN108629799B (en) 2017-03-24 2017-03-24 Method and equipment for realizing augmented reality

Publications (2)

Publication Number Publication Date
CN108629799A CN108629799A (en) 2018-10-09
CN108629799B true CN108629799B (en) 2021-06-01

Family

ID=63707651

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710181831.7A Active CN108629799B (en) 2017-03-24 2017-03-24 Method and equipment for realizing augmented reality

Country Status (1)

Country Link
CN (1) CN108629799B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111325780B (en) * 2020-02-17 2021-07-27 天目爱视(北京)科技有限公司 3D model rapid construction method based on image screening

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103092933A (en) * 2013-01-06 2013-05-08 南京大学 Delaunay triangulation network parallel net-constructing method based on rectangular piecing towards magnanimity point cloud data
CN105023266A (en) * 2014-04-29 2015-11-04 高德软件有限公司 Method and device for implementing augmented reality (AR) and terminal device
CN105258702A (en) * 2015-10-06 2016-01-20 深圳力子机器人有限公司 Global positioning method based on SLAM navigation mobile robot
US20160071327A1 (en) * 2014-09-05 2016-03-10 Fu Tai Hua Industry (Shenzhen) Co., Ltd. System and method for simplifying a mesh point cloud
CN105572687A (en) * 2015-12-11 2016-05-11 中国测绘科学研究院 Method for manufacturing building digital line map based on vehicle-mounted laser radar point cloud
CN105654492A (en) * 2015-12-30 2016-06-08 哈尔滨工业大学 Robust real-time three-dimensional (3D) reconstruction method based on consumer camera

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103092933A (en) * 2013-01-06 2013-05-08 南京大学 Delaunay triangulation network parallel net-constructing method based on rectangular piecing towards magnanimity point cloud data
CN105023266A (en) * 2014-04-29 2015-11-04 高德软件有限公司 Method and device for implementing augmented reality (AR) and terminal device
US20160071327A1 (en) * 2014-09-05 2016-03-10 Fu Tai Hua Industry (Shenzhen) Co., Ltd. System and method for simplifying a mesh point cloud
CN105258702A (en) * 2015-10-06 2016-01-20 深圳力子机器人有限公司 Global positioning method based on SLAM navigation mobile robot
CN105572687A (en) * 2015-12-11 2016-05-11 中国测绘科学研究院 Method for manufacturing building digital line map based on vehicle-mounted laser radar point cloud
CN105654492A (en) * 2015-12-30 2016-06-08 哈尔滨工业大学 Robust real-time three-dimensional (3D) reconstruction method based on consumer camera

Also Published As

Publication number Publication date
CN108629799A (en) 2018-10-09

Similar Documents

Publication Publication Date Title
US10349033B2 (en) Three-dimensional map generating and displaying apparatus and method
KR20170031733A (en) Technologies for adjusting a perspective of a captured image for display
CN106705837B (en) Object measuring method and device based on gestures
JP2011239361A (en) System and method for ar navigation and difference extraction for repeated photographing, and program thereof
US11055534B2 (en) Method and apparatus for 3-D auto tagging
US11037325B2 (en) Information processing apparatus and method of controlling the same
KR101703013B1 (en) 3d scanner and 3d scanning method
US10917622B2 (en) Information processing apparatus, display control method, and storage medium
US10861185B2 (en) Information processing apparatus and method of controlling the same
KR101256046B1 (en) Method and system for body tracking for spatial gesture recognition
CN111783820A (en) Image annotation method and device
CN108629799B (en) Method and equipment for realizing augmented reality
KR20110088995A (en) Method and system to visualize surveillance camera videos within 3d models, and program recording medium
US11108964B2 (en) Information processing apparatus presenting information, information processing method, and storage medium
US10950056B2 (en) Apparatus and method for generating point cloud data
JP6487545B2 (en) Recognition calculation device, recognition calculation method, and recognition calculation program
CN108401452B (en) Apparatus and method for performing real target detection and control using virtual reality head mounted display system
CN111343445A (en) Device and method for dynamically adjusting depth resolution
JP6704712B2 (en) Information processing apparatus, control method of information processing apparatus, and program
JP2017184136A (en) Information processing device, information processing method, information processing system, and program
KR101473234B1 (en) Method and system for displaying an image based on body tracking
US20210258476A1 (en) System for generating a three-dimensional scene of a physical environment
US20200167005A1 (en) Recognition device and recognition method
JP2020095671A (en) Recognition device and recognition method
CN107883930B (en) Pose calculation method and system of display screen

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant