WO2018019272A1 - 基于平面检测实现增强现实的方法及装置 - Google Patents

基于平面检测实现增强现实的方法及装置 Download PDF

Info

Publication number
WO2018019272A1
WO2018019272A1 PCT/CN2017/094736 CN2017094736W WO2018019272A1 WO 2018019272 A1 WO2018019272 A1 WO 2018019272A1 CN 2017094736 W CN2017094736 W CN 2017094736W WO 2018019272 A1 WO2018019272 A1 WO 2018019272A1
Authority
WO
WIPO (PCT)
Prior art keywords
plane
interest
region
video image
image
Prior art date
Application number
PCT/CN2017/094736
Other languages
English (en)
French (fr)
Inventor
柳寅秋
郑黎力
Original Assignee
成都理想境界科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN201610617570.4A external-priority patent/CN107665507B/zh
Priority claimed from CN201610616770.8A external-priority patent/CN107665505B/zh
Application filed by 成都理想境界科技有限公司 filed Critical 成都理想境界科技有限公司
Publication of WO2018019272A1 publication Critical patent/WO2018019272A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics

Definitions

  • the present invention relates to the field of computer vision and artificial intelligence technologies, and in particular, to a method and apparatus for implementing augmented reality based on plane detection.
  • SLAM Simultaneous Localization and Mapping
  • Augmented Reality uses computer graphics technology and visualization technology to generate virtual objects that do not exist in the real environment, and accurately integrates virtual objects into the real environment through image recognition and positioning technology.
  • the real environment is integrated and presented to the user with a realistic sensory experience.
  • the primary technical problem to be solved by augmented reality technology is how to accurately integrate virtual objects into the real world, that is, to make the virtual objects appear in the correct position of the real scene in the correct angle, thus producing a strong visual reality.
  • the augmented reality technology scheme based on the SLAM technology after the environment recognition and the camera pose calculation are completed, the virtual information superimposed display is directly performed without the plane detection, which results in a low degree of fit between the virtual information and the environment scene, and is easy.
  • the user is visually misplaced and affects the rendering effect of the augmented reality technology. Therefore, how to accurately detect the plane of virtual information superposition in different real environment scenes, so that the virtual information is consistent with the plane of the superimposed position is one of the technical problems to be solved in the development of augmented reality technology.
  • An object of the present invention is to provide a method and a device for implementing augmented reality based on plane detection, which can detect a plane in a region of interest in a video image of an environment scene by acquiring a video image of the environment scene in real time, without initial information of the scene or a specific By marking points, the virtual objects can be superimposed and displayed in the region of interest of the video image of the environment scene quickly and accurately.
  • an aspect of the present invention provides a method for implementing augmented reality based on plane detection, including the steps of: acquiring a video image of an environment scene in real time through a camera; and determining a region of interest in a current frame of the video image of the environment scene. Detecting a plane in the region of interest; rendering a virtual object in a current frame of the video image of the environment scene according to a plane in the region of interest.
  • the step of acquiring a video image of the environment scene in real time by the camera further comprising: determining, according to the video image of the environment scene, a camera pose corresponding to a current frame of the video image of the environment scene; A video image of the environmental scene is constructed, and a three-dimensional map of the environmental scene is constructed.
  • the step of determining a region of interest in a current frame of the video image of the environment scene, detecting a plane in the region of interest specifically: determining a video image of the environment scene according to a user manipulation instruction a region of interest in the current frame; or, based on the camera pose, determining a region of interest in a current frame of the video image of the environmental scene; extracting an interest in a current frame of the video image of the environmental scene
  • An image feature point in the region is obtained as a set of image feature points; in the image feature point set, a preset number of image feature points are selected, and a plane in the region of interest is detected.
  • the step of selecting a preset number of image feature points and detecting a plane in the region of interest is specifically: in the image feature point set, by random The sampling consistency algorithm selects a preset number of image feature points; and according to the three-dimensional map of the environment scene, determining corresponding image points of the selected image feature points in the reference frame of the video image of the environment scene, the reference frame a previous key frame or a previous adjacent frame of the current frame of the video image of the environment scene; iteratively calculates a homography matrix according to all selected image feature points and corresponding image points in the reference frame, And determining, according to the homography matrix, a rotation matrix R 0 and a displacement matrix t 0 ; determining, according to a rotation matrix R and a displacement matrix t of the current frame image and the reference frame image, the rotation matrix R 0 and the displacement matrix t 0 Whether the error between the matrix and the matrix including the rotation matrix R and the displacement matrix t is less than a preset
  • the step of determining a region of interest in a current frame of the video image of the environment scene, detecting a plane in the region of interest further comprising: according to the image feature point set to the The image feature points of the planes in the region are less than the image feature points of the preset distance threshold, and the normals of the planes in the region of interest are determined.
  • the step of determining a region of interest in a current frame of the video image of the environment scene, detecting a plane in the region of interest specifically: determining a video image of the environment scene according to a user manipulation instruction a region of interest in the current frame; extracting image feature points in the region of interest in the current frame of the video image of the environment scene, and determining a map point corresponding to the image feature point according to the three-dimensional map of the environment scene Obtaining a map point set; detecting a plane in the region of interest according to the map point set.
  • the step of detecting a plane in the region of interest according to the map point set is specifically: acquiring a three-dimensional map of the environment scene according to an average coordinate value of a map point in the map point set a map point set MP centered on the coordinate origin; convert the map point set MP into a data matrix and perform singular value decomposition to determine a plane P with the singular vector corresponding to the smallest singular value as a normal; recursively calculate the map point The distance between all the map points in the MP to the plane P is determined, and it is determined whether the number of map points whose distance is less than the preset threshold ⁇ is greater than a preset number threshold l, and if yes, determining that the plane P is a plane in the region of interest .
  • the step of rendering a virtual object in a current frame of the video image of the environment scene according to a plane in the region of interest is specifically: adjusting a position and/or an angle of the virtual object, so that Determining a normal direction of the virtual object with a normal direction of a plane in the region of interest, determining a rotation matrix and a displacement matrix of the virtual object; and a region of interest in a current frame of the video image of the environment scene
  • the virtual object is rendered according to a rotation matrix and a displacement matrix of the virtual object.
  • Another aspect of the present invention provides an apparatus for implementing augmented reality based on plane detection, comprising: an image acquisition module: a video image for acquiring an environment scene in real time through a camera; and a plane detection module: a video image for determining the environment scene a region of interest in the current frame, detecting a plane in the region of interest; an object rendering module: for rendering a virtual object in a current frame of the video image of the environment scene according to a plane in the region of interest .
  • the image acquisition module further includes: an image tracking unit configured to: determine a camera pose corresponding to a current frame of the video image of the environment scene according to the video image of the environment scene; and a map construction unit: configured to: Constructing a three-dimensional map of the environmental scene according to the video image of the environmental scene.
  • an image tracking unit configured to: determine a camera pose corresponding to a current frame of the video image of the environment scene according to the video image of the environment scene
  • a map construction unit configured to: Constructing a three-dimensional map of the environmental scene according to the video image of the environmental scene.
  • the plane detecting module further includes: an area determining unit, configured to determine a region of interest in a current frame of the video image of the environment scene according to a user manipulation instruction or the camera pose; the feature point extracting unit The image feature points in the region of interest in the current frame of the video image of the environment scene are extracted to obtain an image feature point set; and the plane detecting unit is configured to select a preset quantity in the image feature point set. Image feature points and detect planes in the region of interest.
  • the plane detecting unit is configured to: in the image feature point set, select a preset number of image feature points by using a random sampling consistency algorithm; and determine each selected one according to the three-dimensional map of the environment scene.
  • the image feature point is a corresponding image point in a reference frame of the video image of the environment scene, and the reference frame is a previous key frame or a previous adjacent frame of a current frame of the video image of the environment scene;
  • Image feature points and corresponding image points in the reference frame iteratively calculates a homography matrix, and determines a rotation matrix R 0 and a displacement matrix t 0 according to the homography matrix; according to the current frame image and the reference frame image a rotation matrix R and a displacement matrix t, determining whether an error between a matrix including the rotation matrix R 0 and the displacement matrix t 0 and a matrix including the rotation matrix R and the displacement matrix t is less than a preset threshold ⁇ , and if so, Then all the selected image feature points are determined to be on the
  • the plane detecting module further includes: a normal determining unit configured to determine, according to an image feature point in the image feature point set that is smaller than a preset distance threshold by a plane to the region in the region of interest The normal to the plane in the region of interest.
  • a normal determining unit configured to determine, according to an image feature point in the image feature point set that is smaller than a preset distance threshold by a plane to the region in the region of interest The normal to the plane in the region of interest.
  • the plane detecting module includes: an area determining unit, configured to determine a region of interest in a current frame of the video image of the environment scene according to a user manipulation command or the camera pose; the feature point extracting unit: And extracting image feature points in the region of interest in the current frame of the video image of the environment scene, determining a map point corresponding to the image feature point according to the three-dimensional map of the environment scene, and obtaining a map point set;
  • the detecting unit is configured to detect a plane in the region of interest according to the set of map points.
  • the plane detecting unit is configured to: acquire, according to an average coordinate value of the map point in the map point set, a map point set MP centered on a coordinate origin of the three-dimensional map of the environment scene; Converting the map point set MP into a data matrix and performing singular value decomposition, determining a plane P with the singular vector corresponding to the smallest singular value as a normal; recursively calculating the distance of all the map points in the map point set MP to the plane P, Determining whether the number of map points whose distance is less than the preset threshold ⁇ is greater than a preset number threshold l, and if so, determining that the plane P is a plane in the region of interest.
  • the object rendering module includes: an object adjusting unit configured to adjust a position and/or an angle of the virtual object such that a normal direction of the virtual object and a normal of a plane in the region of interest Consistent direction, determining a rotation matrix and a displacement matrix of the virtual object; an object rendering unit: a region of interest in a current frame of the video image of the environment scene, rendering according to a rotation matrix and a displacement matrix of the virtual object The virtual object.
  • the invention is based on a method and a device for realizing augmented reality by plane detection.
  • the camera captures the video image of the environment scene in real time, performs camera pose tracking and establishes a three-dimensional map of the environment scene, and determines the user's region of interest in the video image of the environment scene.
  • the plane in the region of interest of the user is detected, the posture adjustment of the virtual object is performed, and the superimposed rendering of the virtual object in the video image of the environment scene is completed.
  • the invention is based on a method and a device for realizing augmented reality by plane detection.
  • the posture of the superimposed rendered virtual object is The planes of the superimposed positions are consistent, which significantly improves the fit of the virtual objects to the real environment scene, and achieves the seamless integration of the virtual objects and the real environment.
  • FIG. 1 is a flow chart showing a method for implementing augmented reality based on plane detection according to a first embodiment of the present invention
  • FIG. 2 is a flow chart showing a method for implementing augmented reality based on plane detection according to a second embodiment of the present invention
  • FIG. 3 is a schematic structural diagram of an apparatus for implementing augmented reality based on plane detection according to a third embodiment of the present invention.
  • FIG. 4 is a schematic structural diagram of an image acquisition module of a device for implementing augmented reality based on plane detection according to a third embodiment of the present invention.
  • FIG. 5 is a schematic structural diagram of a plane detecting module of an apparatus for implementing augmented reality based on plane detection according to a third embodiment of the present invention.
  • FIG. 6 is a schematic structural diagram of an object rendering module of an apparatus for implementing augmented reality based on plane detection according to a third embodiment of the present invention.
  • FIG. 7 is a flow chart showing a method for implementing augmented reality based on plane detection according to a fourth embodiment of the present invention.
  • FIG. 1 is a flow chart showing a method for implementing augmented reality based on plane detection according to a first embodiment of the present invention.
  • a method for implementing augmented reality based on plane detection includes step S101 to Step S103.
  • step S101 the video image of the environmental scene is acquired by the camera in real time.
  • step S102 a region of interest in a current frame of the video image of the environmental scene is determined, and a plane in the region of interest is detected.
  • step S103 a virtual object is rendered in a current frame of the video image of the environmental scene according to a plane in the region of interest.
  • a method for realizing augmented reality based on plane detection is performed, and a video image of an environment scene is acquired by a camera in real time, and a camera pose corresponding to a video image of the environment scene is determined based on simultaneous positioning and map creation technology, and a three-dimensional map of the environment scene is constructed. And, by determining a region of interest of the user in the video image of the environment scene and detecting a plane in the region of interest, superimposed rendering of the virtual object in the current frame of the video image of the environment scene is achieved.
  • FIG. 2 is a flow chart showing a method for implementing augmented reality based on plane detection according to a second embodiment of the present invention.
  • the method for implementing augmented reality based on plane detection includes the following steps S201 to S205.
  • step S201 the video image of the environmental scene is acquired by the camera in real time.
  • step S202 determining a region of interest in a current frame of the video image of the environmental scene according to a user manipulation instruction; or determining a sense in a current frame of the video image of the environmental scene according to the camera pose Area of interest.
  • step S203 image feature points in the region of interest in the current frame of the video image of the environment scene are extracted to obtain an image feature point set.
  • step S204 in the image feature point set, a preset number of image feature points are selected, and a plane in the region of interest is detected.
  • step S205 a virtual object is rendered in a current frame of the video image of the environmental scene according to a plane in the region of interest.
  • the step S201 further includes: determining, according to the video image of the environment scene, a camera pose corresponding to a current frame of the video image of the environment scene; and, according to the video image of the environment scene Constructing a three-dimensional map of the environmental scene.
  • the video image of the environment scene is collected in real time, and the image feature points in the video image of the environment scene are extracted frame by frame to perform video image tracking, and image feature matching is performed on adjacent frames of the video image of the environment scene, according to the minimum re-projection.
  • the error method calculates location and direction information of the camera corresponding to the current frame in the environment scene, and constructs a spatial three-dimensional map describing the environment scene in the three-dimensional space according to the video image of the environment scene, where the three-dimensional map includes the location of the map point And the pose of the camera.
  • the step S202 is specifically: displaying a video image on the screen during the real-time acquisition of the video image of the environmental scene, and determining the touch operation according to the position of the touch operation performed by the user on the screen.
  • the position-centered preset range area is the area of interest in the current frame of the video image of the environment scene, wherein the touch operation may be click, swipe, circle, etc., which is not limited herein;
  • Set the camera position in the time range Whether the amplitude of the change of the posture is less than a preset threshold, and if yes, determining that the central preset range area of the video image of the environmental scene captured by the camera in the preset time range is the interest in the current frame of the video image of the environmental scene region.
  • the step S203 is specifically: extracting image feature points in the region of interest in the current frame of the video image of the environment scene to obtain an image feature point set FP.
  • the step S204 is specifically: first, in the image feature point set FP, a preset number of image feature points are selected by a random sampling consistency algorithm.
  • the number of image feature points selected may be four or more.
  • the corresponding image point of the selected image feature point in the reference frame of the video image of the environment scene is determined. Specifically, the mapping point of the same map point in the video image reference frame of the environment scene is determined according to the selected image feature point corresponding to the map point in the three-dimensional map of the environment scene, that is, the selected image feature point is in the environment scene.
  • the corresponding image point in the reference frame of the video image is specifically: first, in the image feature point set FP, a preset number of image feature points are selected by a random sampling consistency algorithm.
  • the number of image feature points selected may be four or more.
  • the corresponding image point of the selected image feature point in the reference frame of the video image of the environment scene is determined according to the selected image feature point corresponding to the map point in the three-dimensional map of the environment scene
  • the reference frame is a previous key frame or a previous adjacent frame of the current frame of the video image of the environment scene.
  • the homography matrix is iteratively calculated according to the image feature points and the corresponding image points of the image feature points in the reference frame, and the rotation matrix R 0 and the displacement matrix t 0 are determined accordingly. Then, according to the rotation matrix R and the displacement matrix t between the current frame and the reference frame determined in the video image tracking process of the environment scene, determining a matrix including the rotation matrix R 0 and the displacement matrix t 0 and including the rotation matrix Whether the error between the matrix of R and the displacement matrix t is less than a preset threshold ⁇ .
  • the step S204 further includes: determining, according to the image feature points in the image feature point set FP that the distance to the plane P 0 in the region of interest is less than a preset distance threshold, determining the region of interest The normal of the plane P 0 .
  • the step S205 is specifically: adjusting a position and/or an angle of the virtual object such that a normal direction of the virtual object is consistent with a normal direction of a plane in the region of interest, Determining a rotation matrix and a displacement matrix of the virtual object; rendering the virtual object according to a rotation matrix and a displacement matrix of the virtual object in a region of interest in a current frame of the video image of the environment scene.
  • the center of the region of interest in the video image of the environment scene is a rendering position of the virtual object, and the position and/or angle of the virtual object is adjusted to make the normal direction of the virtual object and the sense
  • the normal directions of the planes in the region of interest are identical, the rotation matrix and the displacement matrix of the virtual object are determined, and the virtual object is rendered accordingly.
  • the video image of the environment scene is collected by the camera in real time, the camera pose is tracked and a three-dimensional map of the environment scene is established, and the user interested area in the video image of the environment scene is determined.
  • Performing posture adjustment of the virtual object by detecting a plane in the region of interest of the user, and completing Overlay rendering of a virtual object in a video image of an environment scene.
  • the embodiment of the invention is based on the sparse point cloud to detect the plane in the user's region of interest in the environment scene, and has high computational efficiency, and is suitable for the implementation of the mobile augmented reality, and can simultaneously make the posture of the superimposed and rendered virtual object consistent with the plane of the superimposed position, and significant Improve the fit of the virtual object and the real environment scene, and realize the seamless integration of the virtual object and the real environment.
  • FIG. 3 is a block diagram showing the structure of an apparatus for realizing augmented reality based on plane detection according to a third embodiment of the present invention.
  • the apparatus 300 for implementing augmented reality based on the plane detection mainly includes: an image acquisition module 301: a video image for acquiring an environment scene in real time through a camera; and a plane detection module 302: for determining The region of interest in the current frame of the video image of the environment scene acquired by the image acquisition module 301, detecting a plane in the region of interest; the object rendering module 303: configured to detect according to the plane detection module 302 A plane in the region of interest that renders the virtual object in the current frame of the video image of the environment scene.
  • the image acquisition module 301 acquires the video image of the environmental scene in real time through the camera, determines the camera pose corresponding to the video image of the environmental scene based on the simultaneous positioning and map creation technology, and constructs a three-dimensional map of the environmental scene.
  • the plane detection module 302 determines a region of interest of the user in the video image of the environmental scene and detects a plane in the region of interest.
  • the object rendering module 303 implements overlay rendering of the virtual object in the current frame of the video image of the environmental scene based on the plane in the region of interest.
  • the image capturing module 301 further includes: an image tracking unit 3011: configured to determine a video image of the environmental scene according to a video image of the environmental scene a camera pose corresponding to the current frame; a map construction unit 3012: configured to construct a three-dimensional map of the environment scene according to the video image of the environment scene.
  • the image tracking unit 3011 performs video image tracking by extracting image feature points in the video image of the environment scene frame by frame, and performs image feature matching on adjacent frames of the video image of the environment scene, by minimizing the re-projection error.
  • the method calculates the position and direction information of the camera corresponding to the current frame in the environment scene.
  • the map construction unit 3012 constructs a spatial three-dimensional map describing the environmental scene in a three-dimensional space according to the video image of the environmental scene, where the three-dimensional map includes the position of the map point and the pose of the camera.
  • the plane detecting module 302 includes: an area determining unit 3021: configured to determine a current video image of the environmental scene according to a user manipulation instruction or the camera pose a region of interest in the frame; a feature point extraction unit 3022: for extracting image feature points in the region of interest in the current frame of the video image of the environment scene, to obtain an image feature point set; and a plane detecting unit 3023: for In the stated In the image feature point set, a preset number of image feature points are selected, and a plane in the region of interest is detected.
  • the image acquisition module 301 displays a video image on the screen during the real-time acquisition of the video image of the environmental scene.
  • the area determining unit 3021 determines, according to the location where the user performs the touch operation on the screen, the preset range area centered on the touch operation position is the interest area in the current frame of the video image of the environment scene,
  • the touch operation may be a click, a swipe, a circle, or the like, which is not limited herein.
  • the area determining unit 3021 determines whether the variation range of the camera pose in the preset time range is less than a preset threshold, and if yes, determines a center preset range of the video image of the environment scene captured by the camera in the preset time range.
  • the region is the region of interest in the current frame of the video image of the environmental scene.
  • the feature point extracting unit 3022 extracts image feature points in the region of interest in the current frame of the video image of the environment scene to obtain an image feature point set FP.
  • the plane detecting unit 3023 selects, in the image feature point set FP, a preset number of image feature points by using a random sampling consistency algorithm, wherein the selected number of image feature points may be 4 Or more than 4; according to the three-dimensional map of the environment scene, determining the corresponding image point of the selected image feature point in the reference frame of the video image of the environment scene, specifically, according to the selected image feature point in the three-dimensional environment scene Corresponding map points in the map, determining mapping points of the same map point in the video image reference frame of the environment scene, that is, corresponding image points of the selected image feature points in the reference frame of the video image of the environment scene, wherein, The frame is a previous key frame or a previous adjacent frame of the current frame of the video image of the environment scene; iteratively calculates the homography according to the image feature point and the corresponding image point of the image feature point in the reference frame matrix, and accordingly determining the rotation matrix R 0 and displacement matrix t 0; when tracking
  • the plane detecting module 302 further includes: a normal determining unit (not shown in the drawing), configured to use, according to the image feature point set FP, a distance to a plane P 0 in the region of interest is smaller than The image feature points of the preset threshold determine the normal of the plane P 0 in the region of interest.
  • a normal determining unit (not shown in the drawing), configured to use, according to the image feature point set FP, a distance to a plane P 0 in the region of interest is smaller than The image feature points of the preset threshold determine the normal of the plane P 0 in the region of interest.
  • the object rendering module 303 includes: an object adjusting unit 3031: a method for adjusting a position and/or an angle of the virtual object, and making the virtual object a line direction is consistent with a normal direction of a plane in the region of interest, determining a rotation matrix and a displacement matrix of the virtual object; and an object rendering unit 3032: for sensing in a current frame of the video image of the environment scene Area of interest, according to the virtual The virtual matrix is rendered by the object's rotation matrix and displacement matrix.
  • the object adjustment unit 3031 adjusts the position and/or angle of the virtual object with the center of the region of interest in the video image of the environment scene as a rendering position of the virtual object, so that the normal direction of the virtual object Consistent with a normal direction of a plane in the region of interest, determining a rotation matrix and a displacement matrix of the virtual object; an object rendering unit 3032, a video image of the environment scene according to a rotation matrix and a displacement matrix of the virtual object The virtual object is rendered within the region of interest in the current frame.
  • the device for realizing augmented reality based on the plane detection in the embodiment of the present invention collects the video image of the environment scene in real time through the camera, performs camera pose tracking and establishes a three-dimensional map of the environment scene, and determines the user interested area in the video image of the environment scene. By detecting the plane in the region of interest of the user, the posture adjustment of the virtual object is performed, and the superimposed rendering of the virtual object in the video image of the environment scene is completed.
  • the embodiment of the present invention detects a plane in a region of interest of a user in an environment scene based on a sparse point cloud, and has a small calculation amount and high computational efficiency, and is suitable for the implementation of the mobile augmented reality, and can simultaneously make the posture of the superimposed rendered virtual object and the plane of the superimposed position thereof. Consistently, the fit of the virtual object to the real environment scene is significantly improved, and the effect of the virtual object and the real environment is seamlessly integrated.
  • FIG. 7 is a flow chart showing a method for implementing augmented reality based on plane detection according to a fourth embodiment of the present invention.
  • the method for implementing the augmented reality based on the plane detection in the embodiment of the present invention includes the following steps: Step S401: acquiring a video image of an environment scene in real time by using a camera; and step S402, determining the environment scenario according to a user manipulation instruction.
  • step S403 extracting image feature points in the region of interest in the current frame of the video image of the environment scene, and determining the image according to the three-dimensional map of the environment scene a map point corresponding to the feature point, obtaining a map point set; step S404, detecting a plane in the region of interest according to the map point set; step S405, according to a plane in the region of interest, in the environment scene
  • the virtual object is rendered in the current frame of the video image.
  • the step S401 further includes: determining, according to the video image of the environment scene, a camera pose corresponding to a current frame of the video image of the environment scene; and, according to the video image of the environment scene Constructing a three-dimensional map of the environmental scene.
  • the video image of the environment scene is collected in real time, and the image feature points in the video image of the environment scene are extracted frame by frame, and the image features of the adjacent frames of the video image of the environment scene are matched, and the method according to the method of minimizing the re-projection error is calculated.
  • the position and direction information of the camera corresponding to the current frame is located in the environment scene, and the spatial three-dimensional map describing the environment scene is constructed in the three-dimensional space according to the video image of the environment scene, where the location of the map point and the position of the camera are included in the three-dimensional map. posture.
  • the step S402 is specifically: during the real-time collection process of the video image of the environment scene, Displaying a video image on the screen, and determining, according to a location where the user performs a touch operation on the screen, a preset range area centered on the touch operation position is a region of interest in a current frame of the video image of the environment scene
  • the touch operation may be click, swipe, circle, etc., and is not limited herein; or, according to the user's manipulation instruction, the camera focuses on a scene area in the environment scene that the user wishes to render the virtual object, and the scene area includes a plane. Or an approximate plane, such as a desktop, a wall or a ground, etc., determining that the scene area is a region of interest in a current frame of a video image of the environmental scene.
  • the step S403 is specifically: extracting image feature points in the region of interest in the current frame of the video image of the environment scene, and determining the image feature according to the three-dimensional map of the environment scene. Click the corresponding map point to get the map point set FP.
  • the step S404 is specifically: subtracting, according to the average coordinate value of the map points in the map point set FP, the coordinate value of each map point in the map point set FP by the average coordinate a value, obtaining a map point set MP centered on a coordinate origin of the three-dimensional map of the environment scene; converting the map point set MP into a data matrix and performing singular value decomposition to determine a plane having a minimum singular value vector as a normal P; recursively calculating the distance of all the map points in the map point set MP to the plane P, determining whether the number of map points whose distance is less than the preset threshold ⁇ is greater than a preset number threshold l, and if yes, determining that the plane P is A plane in the region of interest.
  • the step S405 is specifically: adjusting a position and/or an angle of the virtual object such that a normal direction of the virtual object is consistent with a normal direction of a plane in the region of interest, Determining a rotation matrix and a displacement matrix of the virtual object; rendering the virtual object according to a rotation matrix and a displacement matrix of the virtual object in a region of interest in a current frame of the video image of the environment scene.
  • the center of the region of interest in the video image of the environment scene is a rendering position of the virtual object, and the position and/or angle of the virtual object is adjusted to make the normal direction of the virtual object and the sense
  • the normal directions of the planes in the region of interest are identical, the rotation matrix and the displacement matrix of the virtual object are determined, and the virtual object is rendered accordingly.
  • the method for realizing augmented reality based on plane detection in the embodiment of the present invention, the video image of the environment scene is collected by the camera in real time, the camera pose is tracked and a three-dimensional map of the environment scene is established, and the user interested area in the video image of the environment scene is determined.
  • the posture adjustment of the virtual object is performed, and the superimposed rendering of the virtual object in the video image of the environment scene is completed.
  • the embodiment of the invention is based on the sparse point cloud to detect the plane in the user's region of interest in the environment scene, and has high computational efficiency, and is suitable for the implementation of the mobile augmented reality, and can simultaneously make the posture of the superimposed and rendered virtual object consistent with the plane of the superimposed position, and significant Improve the fit of the virtual object and the real environment scene, and realize the seamless integration of the virtual object and the real environment.
  • This embodiment relates to a schematic structural diagram of an apparatus for realizing augmented reality based on plane detection. Since the structural schematic of the apparatus of the present embodiment is similar to that of the apparatus described in FIGS. 3 to 6, the drawings will not be separately described herein.
  • the method includes: an image acquisition module 301: a video image for acquiring an environment scene in real time through a camera; a plane detection module 302: a current frame for determining a video image of an environment scene acquired by the image acquisition module 301 a region of interest, detecting a plane in the region of interest; an object rendering module 303: a video image for the environment scene according to a plane in the region of interest detected by the plane detection module 302 Renders a virtual object in the current frame.
  • the image acquisition module 301 acquires the video image of the environmental scene in real time through the camera, determines the camera pose corresponding to the video image of the environmental scene based on the simultaneous positioning and map creation technology, and constructs a three-dimensional map of the environmental scene.
  • the plane detection module 302 determines a user region of interest in the video image of the environment scene and detects a plane in the region of interest, and the object rendering module 303 implements the virtual object in the current frame of the video image of the environment scene according to the plane in the region of interest. Overlay rendering.
  • the image capturing module 301 further includes: an image tracking unit 3011: configured to determine a video image of the environmental scene according to a video image of the environmental scene a camera pose corresponding to the current frame; a map construction unit 3012: configured to construct a three-dimensional map of the environment scene according to the video image of the environment scene.
  • the image tracking unit 3011 performs image feature matching on adjacent frames of the video image of the environmental scene by extracting image feature points in the video image of the environment scene frame by frame, and calculates the current method by minimizing the re-projection error.
  • the position and direction information of the camera corresponding to the frame is located in the environment scene;
  • the map construction unit 3012 constructs a spatial three-dimensional map describing the environment scene in the three-dimensional space according to the video image of the environment scene, where the three-dimensional map includes the location of the map point and the camera Position.
  • the plane detecting module 302 includes: an area determining unit 3021: configured to determine a region of interest in a current frame of a video image of the environment scene according to a user manipulation instruction.
  • Feature point extraction unit 3022 for extracting image feature points in the region of interest in the current frame of the video image of the environment scene, and determining a map point corresponding to the image feature point according to the three-dimensional map of the environment scene Obtaining a map point set; the plane detecting unit 3023 is configured to detect a plane in the region of interest according to the map point set.
  • the image capturing module 301 performs a video image real-time acquisition process of the environment scene, and displays a video image on the screen, and the area determining unit 3021 performs a touch operation according to the user on the screen.
  • Position, the preset range area centered on the touch operation position is a region of interest in a current frame of the video image of the environment scene, where the touch operation may be click, swipe, circle, and the like.
  • the area determining unit 3021 focuses the camera on a scene area in the environment scene where the user wishes to render the virtual object, the scene area including a plane or an approximate plane, such as a desktop, a wall or a ground, etc., according to a user manipulation instruction. Determining that the scene area is a region of interest in a current frame of a video image of the environmental scene.
  • the feature point extracting unit 3022 extracts image feature points in the region of interest in the current frame of the video image of the environment scene, and determines the image according to the three-dimensional map of the environment scene. A map point corresponding to the map point, and a map point set FP is obtained.
  • the plane detecting unit 3023 subtracts the coordinate value of each map point in the map point set from the average coordinate value according to the average coordinate value of the map point in the map point set FP.
  • the object rendering module 303 includes: an object adjusting unit 3031: a method for adjusting a position and/or an angle of the virtual object, and making the virtual object a line direction is consistent with a normal direction of a plane in the region of interest, determining a rotation matrix and a displacement matrix of the virtual object; and an object rendering unit 3032: for sensing in a current frame of the video image of the environment scene a region of interest, rendering the virtual object according to a rotation matrix and a displacement matrix of the virtual object.
  • the object adjustment unit 3031 adjusts the position and/or angle of the virtual object with the center of the region of interest in the video image of the environment scene as a rendering position of the virtual object, so that the normal direction of the virtual object Consistent with a normal direction of a plane in the region of interest, determining a rotation matrix and a displacement matrix of the virtual object; an object rendering unit 3032, a video image of the environment scene according to a rotation matrix and a displacement matrix of the virtual object The virtual object is rendered within the region of interest in the current frame.
  • the device for realizing augmented reality based on the plane detection in the embodiment of the present invention collects the video image of the environment scene in real time through the camera, performs camera pose tracking and establishes a three-dimensional map of the environment scene, and determines the user interested area in the video image of the environment scene. By detecting the plane in the region of interest of the user, the posture adjustment of the virtual object is performed, and the superimposed rendering of the virtual object in the video image of the environment scene is completed.
  • the embodiment of the present invention detects a plane in a region of interest of a user in an environment scene based on a sparse point cloud, and has a small calculation amount and high computational efficiency, and is suitable for the implementation of the mobile augmented reality, and can simultaneously make the posture of the superimposed rendered virtual object and the plane of the superimposed position thereof. Consistently, the fit of the virtual object to the real environment scene is significantly improved, and the effect of the virtual object and the real environment is seamlessly integrated.
  • the invention is not limited to the specific embodiments described above.
  • the invention can be extended to any new feature or any new combination disclosed in this specification, as well as any novel method or process steps or any new combination disclosed.

Abstract

本发明提供一种基于平面检测实现增强现实的方法，包括以下步骤：通过摄像机实时获取环境场景的视频图像；确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。本发明同时提供一种基于平面检测实现增强现实的装置。本发明基于平面检测实现增强现实的方法及装置，通过检测环境场景的视频图像的用户感兴趣区域中的平面，使叠加渲染的虚拟对象的姿态与其叠加位置的平面保持一致，显著提升了虚拟对象与真实环境场景的贴合度，实现虚拟对象与真实环境的无缝融合。

Description

基于平面检测实现增强现实的方法及装置
本申请要求享有2016年7月29日提交的名称为“基于平面检测实现增强现实的方法及装置”的中国专利申请CN201610617570.4、以及2016年7月29日提交的名称为“基于平面检测实现增强现实的方法及装置”的中国专利申请CN201610616770.8的优先权，其全部内容通过引用并入本文中。
技术领域
本发明涉及计算机视觉与人工智能技术领域，尤其涉及一种基于平面检测实现增强现实的方法及装置。
背景技术
SLAM(Simultaneous Localization and Mapping，同时定位与地图创建)，是一项应用于在未知环境中自主地图创建与自身定位相结合的技术，是当前自主导航领域的研究热点。该技术的目标是解决在进入未知环境后，如何感知周围环境并构建增量式地图，同时进行自身定位的问题，目前主要应用于无人驾驶、机器人及场景三维重建等技术方向。
增强现实技术(Augmented Reality，AR)借助计算机图形技术和可视化技术生成现实环境中不存在的虚拟对象，并通过图像识别定位技术将虚拟对象准确地融合到真实环境中，借助显示设备将虚拟对象与真实环境融为一体，并呈现给使用者真实的感观体验。增强现实技术要解决的首要技术难题是如何将虚拟对象准确地融合到真实世界中，也就是要使虚拟对象以正确的角度姿态出现在真实场景的正确位置上，从而产生强烈的视觉真实感。
现有技术中基于SLAM技术的增强现实技术方案，完成环境识别与摄像机位姿计算之后，未进行平面检测便直接进行虚拟信息叠加显示，这导致虚拟信息与环境场景的贴合度较低，容易使用户产生视觉错位，影响增强现实技术的呈现效果。因此，如何在不同的真实环境场景中准确检测虚拟信息叠加的平面，使虚拟信息与其叠加位置的平面保持一致，是增强现实技术发展中亟待解决的技术问题之一。
发明内容
本发明的目的是提供一种基于平面检测实现增强现实的方法及装置，通过实时采集环境场景的视频图像，检测环境场景的视频图像中感兴趣区域中的平面，无需场景的初始信息或者特定的标记点，即可快速、准确地实现在环境场景的视频图像的感兴趣区域中对虚拟对象进行叠加显示。
有鉴于此，本发明一方面提供一种基于平面检测实现增强现实的方法，包括以下步骤：通过摄像机实时获取环境场景的视频图像；确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
优选地，所述通过摄像机实时获取环境场景的视频图像的步骤，还包括：根据所述环境场景的视频图像，确定所述环境场景的视频图像的当前帧对应的摄像机位姿；以及，根据所述环境场景的视频图像，构建所述环境场景的三维地图。
优选地，所述确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面的步骤，具体为：根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；或者，根据所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域；提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合；在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面。
优选地，所述在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面的步骤，具体为：在所述图像特征点集合中，通过随机抽样一致性算法选取预设数量的图像特征点；根据所述环境场景的三维地图，确定各个选取的图像特征点在所述环境场景的视频图像的参考帧中的对应图像点，所述参考帧为所述环境场景的视频图像的当前帧的前一关键帧或者前一相邻帧；根据所有选取的图像特征点及其在所述参考帧中的对应图像点，迭代计算单应性矩阵，并根据所述单应性矩阵确定旋转矩阵R0和位移矩阵t0；根据当前帧图像与参考帧图像的旋转矩阵R和位移矩阵t，判断包括所述旋转矩阵R0和位移矩阵t0的矩阵与包括所述旋转矩阵R和位移矩阵t的矩阵之间的误差是否小于预设阈值ε，若是，则确定所有选取的图像特征点在同一平面上，并确定该平面为所述感兴趣区域中的平面。
优选地，所述确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面的步骤，还包括：根据所述图像特征点集合中到所述感兴趣区域中的平面的距离小于预设距离阈值的图像特征点，确定所述感兴趣区域中的平面的法线。
优选地，所述确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面的步骤，具体为：根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合；根据所述地图点集合，检测所述感兴趣区域中的平面。
优选地，所述根据所述地图点集合，检测所述感兴趣区域中的平面的步骤，具体为：根据所述地图点集合中地图点的平均坐标值，获取以所述环境场景的三维地图的坐标原点为中心的地图点集合MP；将所述地图点集合MP转换为数据矩阵并进行奇异值分解，确定以最小奇异值对应的奇异向量为法线的平面P；递归计算所述地图点集合MP中所有地图点到所述平面P的距离，判断距离小于预设阈值ε的地图点数量是否大于预设数量阈值l，若是，则确定所述平面P为所述感兴趣区域中的平面。
优选地，所述根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象的步骤，具体为：调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；在所述环境场景的视频图像的当前帧中的感兴趣区域内，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。
本发明另一方面提供一种基于平面检测实现增强现实的装置，其包括：图像采集模块：用于通过摄像机实时获取环境场景的视频图像；平面检测模块：用于确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；对象渲染模块：用于根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
优选地，所述图像采集模块，还包括：图像跟踪单元：用于根据所述环境场景的视频图像，确定所述环境场景的视频图像的当前帧对应的摄像机位姿；地图构建单元：用于根据所述环境场景的视频图像，构建所述环境场景的三维地图。
优选地，所述平面检测模块，还包括：区域确定单元：用于根据用户操控指令或所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域；特征点提取单元：用于提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合；平面检测单元：用于在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面。
优选地，所述平面检测单元，具体用于：在所述图像特征点集合中，通过随机抽样一致性算法选取预设数量的图像特征点；根据所述环境场景的三维地图，确定各个选取的图 像特征点在所述环境场景的视频图像的参考帧中的对应图像点，所述参考帧为所述环境场景的视频图像的当前帧的前一关键帧或者前一相邻帧；根据所有选取图像特征点及其在所述参考帧中的对应图像点，迭代计算单应性矩阵，并根据所述单应性矩阵确定旋转矩阵R0和位移矩阵t0；根据当前帧图像与参考帧图像的旋转矩阵R和位移矩阵t，判断包括所述旋转矩阵R0和位移矩阵t0的矩阵与包括所述旋转矩阵R和位移矩阵t的矩阵之间的误差是否小于预设阈值ε，若是，则确定所有选取的图像特征点在同一平面上，并确定该平面为所述感兴趣区域中的平面。
优选地，所述平面检测模块，还包括：法线确定单元：用于根据所述图像特征点集合中到所述感兴趣区域中的平面的距离小于预设距离阈值的图像特征点，确定所述感兴趣区域中的平面的法线。
优选地，所述平面检测模块，包括：区域确定单元：用于根据用户操控指令或所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域；特征点提取单元：用于提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合；平面检测单元：用于根据所述地图点集合，检测所述感兴趣区域中的平面。
优选地，所述平面检测单元，具体用于：根据所述地图点集合中地图点的平均坐标值，获取以所述环境场景的三维地图的坐标原点为中心的地图点集合MP；将所述地图点集合MP转换为数据矩阵并进行奇异值分解，确定以最小奇异值对应的奇异向量为法线的平面P；递归计算所述地图点集合MP中所有地图点到所述平面P的距离，判断距离小于预设阈值ε的地图点数量是否大于预设数量阈值l，若是，则确定所述平面P为所述感兴趣区域中的平面。
优选地，所述对象渲染模块，包括：对象调整单元：用于调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；对象渲染单元：用于在所述环境场景的视频图像的当前帧中的感兴趣区域，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。
本发明基于平面检测实现增强现实的方法及装置，通过摄像机实时采集环境场景的视频图像，进行摄像机位姿跟踪并建立环境场景的三维地图，确定环境场景的视频图像中的用户感兴趣区域，通过检测用户感兴趣区域中的平面，进行虚拟对象的姿态调整，完成虚拟对象在环境场景的视频图像中的叠加渲染。本发明基于平面检测实现增强现实的方法及装置，通过检测环境场景中用户感兴趣区域中的平面，使叠加渲染的虚拟对象的姿态与其 叠加位置的平面保持一致，显著提升了虚拟对象与真实环境场景的贴合度，实现虚拟对象与真实环境无缝融合的效果。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图：
图1示出了本发明第一实施例的基于平面检测实现增强现实的方法的流程示意图；
图2示出了本发明第二实施例的基于平面检测实现增强现实的方法的流程示意图；
图3示出了本发明第三实施例的基于平面检测实现增强现实的装置的结构示意图；
图4示出了本发明第三实施例的基于平面检测实现增强现实的装置的图像采集模块的结构示意图；
图5示出了本发明第三实施例的基于平面检测实现增强现实的装置的平面检测模块的结构示意图；
图6示出了本发明第三实施例的基于平面检测实现增强现实的装置的对象渲染模块的结构示意图；
图7示出了本发明第四实施例的基于平面检测实现增强现实的方法的流程示意图。
具体实施方式
为了能够更清楚地理解本发明的目的、特征和优点，下面结合附图和具体实施方式对本发明做进一步的详细描述。需要说明的是，在不冲突的情况下，本申请的实施例及实施例中的特征可以相互结合。
在下面的描述中阐述了很多具体细节以便于充分理解本发明，但是，这仅仅是本发明的一些实施例，本发明还可以采用其他不同于在此描述的其他方式来实施，因此，本发明的保护范围并不受下面公开的具体实施例的限制。
实施例一
图1示出了本发明第一实施例的基于平面检测实现增强现实的方法的流程示意图。
本发明实施例的基于平面检测实现增强现实的方法，如图1所示，包括步骤S101至 步骤S103。其中，在步骤S101中，通过摄像机实时获取环境场景的视频图像。在步骤S102中，确定所述环境场景的视频图像的当前帧中的感兴趣区域，并检测所述感兴趣区域中的平面。在步骤S103中，根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
本发明实施例基于平面检测实现增强现实的方法，通过摄像机实时获取环境场景的视频图像，基于同时定位与地图创建技术，确定环境场景的视频图像对应的摄像机位姿并构建环境场景的三维地图，以及，通过确定环境场景的视频图像中的用户感兴趣区域并检测感兴趣区域中的平面，实现虚拟对象在环境场景的视频图像的当前帧中的叠加渲染。
实施例二
图2示出了本发明第二实施例的基于平面检测实现增强现实的方法的流程示意图。
本发明实施例的基于平面检测实现增强现实的方法，如图2所示，包括以下步骤S201至步骤S205。其中，在步骤S201中，通过摄像机实时获取环境场景的视频图像。在步骤S202中，根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；或者，根据所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域。在步骤S203中，提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合。在步骤S204中，在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面。在步骤S205中，根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
在上述技术方案中，所述步骤S201，还包括：根据所述环境场景的视频图像，确定所述环境场景的视频图像的当前帧对应的摄像机位姿；以及，根据所述环境场景的视频图像，构建所述环境场景的三维地图。具体地，实时采集环境场景的视频图像，通过逐帧提取环境场景的视频图像中的图像特征点进行视频图像跟踪，对环境场景的视频图像的相邻帧进行图像特征匹配，根据最小化重投影误差的方法计算当前帧对应的摄像机位于环境场景中的位置和方向信息，以及，根据环境场景的视频图像在三维空间中构建描述环境场景的空间三维地图，所述三维地图中包括地图点的位置和摄像机的位姿。
在上述技术方案，所述步骤S202，具体为：在环境场景的视频图像实时采集过程中，在屏幕中显示视频图像，根据用户在屏幕上进行触控操作的位置，确定以所述触控操作位置为中心的预设范围区域为所述环境场景的视频图像的当前帧中的感兴趣区域，其中，触控操作可以是点击、划动、圈选等，此处不作限定；或者，判断预设时间范围内摄像机位 姿的变化幅度是否小于预设阈值，若是，则确定所述预设时间范围内摄像机采集的环境场景的视频图像的中心预设范围区域为所述环境场景的视频图像的当前帧中的感兴趣区域。
在上述技术方案中，所述步骤S203，具体为：提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合FP。
在上述技术方案中，所述步骤S204，具体为：首先，在所述图像特征点集合FP中，通过随机抽样一致性算法选取预设数量的图像特征点。其中，图像特征点的选取数量可以是4个或4个以上。其次，根据环境场景的三维地图，确定所选取的图像特征点在环境场景的视频图像的参考帧中的对应图像点。具体地，根据所选取的图像特征点在环境场景的三维地图中对应的地图点，确定相同地图点在环境场景的视频图像参考帧中的映射点，即为所选取的图像特征点在环境场景的视频图像的参考帧中的对应图像点。其中，参考帧为所述环境场景的视频图像的当前帧的前一关键帧或者前一相邻帧。再次，根据图像特征点以及所述图像特征点在所述参考帧中的对应图像点，迭代计算单应性矩阵，并据此确定旋转矩阵R0和位移矩阵t0。然后，根据环境场景的视频图像跟踪过程中确定的当前帧与参考帧之间的旋转矩阵R和位移矩阵t，判断包括所述旋转矩阵R0和位移矩阵t0的矩阵与包括所述旋转矩阵R和位移矩阵t的矩阵之间的误差是否小于预设阈值ε。若是，则确定所有选取的图像特征点在同一平面P0上，并确定该平面P0为所述感兴趣区域中的平面。否则，则确定所述当前帧图像中不存在符合虚拟对象叠加渲染条件的平面。
在上述技术方案中，所述步骤S204，还包括：根据所述图像特征点集合FP中到感兴趣区域中的平面P0的距离小于预设距离阈值的图像特征点，确定感兴趣区域中的平面P0的法线。
在上述技术方案中，所述步骤S205，具体为：调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；在所述环境场景的视频图像的当前帧中的感兴趣区域，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。具体地，以所述环境场景的视频图像中的感兴趣区域的中心为虚拟对象的渲染位置，调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵，并据此渲染所述虚拟对象。
本发明实施例的基于平面检测实现增强现实的方法，通过摄像机实时采集环境场景的视频图像，进行摄像机位姿跟踪并建立环境场景的三维地图，确定环境场景的视频图像中的用户感兴趣区域，通过检测用户感兴趣区域中的平面，进行虚拟对象的姿态调整，完成 虚拟对象在环境场景的视频图像中的叠加渲染。本发明实施例基于稀疏点云检测环境场景中用户感兴趣区域中的平面，运算效率高，适用于移动增强现实实现，同时能够使叠加渲染的虚拟对象的姿态与其叠加位置的平面保持一致，显著提升了虚拟对象与真实环境场景的贴合度，实现虚拟对象与真实环境无缝融合的效果。
实施例三
图3示出了本发明第三实施例的基于平面检测实现增强现实的装置的结构示意图。
本发明实施例的基于平面检测实现增强现实的装置300，如图3所示，主要包括：图像采集模块301：用于通过摄像机实时获取环境场景的视频图像；平面检测模块302：用于确定所述图像采集模块301获取的环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；对象渲染模块303：用于根据所述平面检测模块302检测到的所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
在该技术方案中，图像采集模块301通过摄像机实时获取环境场景的视频图像，基于同时定位与地图创建技术，确定环境场景的视频图像对应的摄像机位姿并构建环境场景的三维地图。平面检测模块302确定环境场景的视频图像中的用户感兴趣区域并检测感兴趣区域中的平面。对象渲染模块303根据感兴趣区域中的平面实现虚拟对象在环境场景的视频图像的当前帧中的叠加渲染。
在上述技术方案中，优选地，所述图像采集模块301，如图4所示，还包括：图像跟踪单元3011：用于根据所述环境场景的视频图像，确定所述环境场景的视频图像的当前帧对应的摄像机位姿；地图构建单元3012：用于根据所述环境场景的视频图像，构建所述环境场景的三维地图。
在该技术方案中，图像跟踪单元3011通过逐帧提取环境场景的视频图像中的图像特征点进行视频图像跟踪，对环境场景的视频图像的相邻帧进行图像特征匹配，通过最小化重投影误差的方法计算当前帧对应的摄像机位于环境场景中的位置和方向信息。地图构建单元3012根据环境场景的视频图像在三维空间中构建描述环境场景的空间三维地图，所述三维地图中包括地图点的位置和摄像机的位姿。
在上述技术方案中，优选地，平面检测模块302，如图5所示，包括：区域确定单元3021：用于根据用户操控指令或所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域；特征点提取单元3022：用于提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合；平面检测单元3023：用于在所述 图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面。
在该技术方案中，具体地，图像采集模块301进行环境场景的视频图像实时采集过程中，在屏幕中显示视频图像。所述区域确定单元3021根据用户在屏幕上进行触控操作的位置，确定以所述触控操作位置为中心的预设范围区域为所述环境场景的视频图像的当前帧中的感兴趣区域，其中，触控操作可以是点击、划动、圈选等，此处不作限定。或者，所述区域确定单元3021判断预设时间范围内摄像机位姿的变化幅度是否小于预设阈值，若是，则确定所述预设时间范围内摄像机采集的环境场景的视频图像的中心预设范围区域为所述环境场景的视频图像的当前帧中的感兴趣区域。
在上述技术方案中，具体地，特征点提取单元3022，提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合FP。
在上述技术方案中，具体地，平面检测单元3023，在所述图像特征点集合FP中，通过随机抽样一致性算法选取预设数量的图像特征点，其中，图像特征点的选取数量可以是4个或4个以上；根据环境场景的三维地图，确定所选取的图像特征点在环境场景的视频图像的参考帧中的对应图像点，具体地，根据所选取的图像特征点在环境场景的三维地图中对应的地图点，确定相同地图点在环境场景的视频图像参考帧中的映射点，即为所选取的图像特征点在环境场景的视频图像的参考帧中的对应图像点，其中，参考帧为所述环境场景的视频图像的当前帧的前一关键帧或者前一相邻帧；根据图像特征点以及所述图像特征点在所述参考帧中的对应图像点，迭代计算单应性矩阵，并据此确定旋转矩阵R0和位移矩阵t0；根据环境场景的视频图像跟踪过程中确定的所述当前帧与参考帧之间的旋转矩阵R和位移矩阵t，判断包括所述旋转矩阵R0和位移矩阵t0的矩阵与包括所述旋转矩阵R和位移矩阵t的矩阵之间的误差是否小于预设阈值ε，若是，则确定所有选取的图像特征点在同一平面P0上，并确定所述平面P0为所述感兴趣区域中的平面。否则，则确定所述当前帧图像中不存在符合虚拟对象叠加渲染条件的平面。
在上述技术方案中，平面检测模块302，还包括：法线确定单元(附图中未示出)，用于根据所述图像特征点集合FP中到感兴趣区域中的平面P0的距离小于预设阈值的图像特征点，确定感兴趣区域中的平面P0的法线。
在上述技术方案中，优选地，所述对象渲染模块303，如图6所示，包括：对象调整单元3031：用于调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；对象渲染单元3032：用于在所述环境场景的视频图像的当前帧中的感兴趣区域，根据所述虚拟 对象的旋转矩阵和位移矩阵渲染所述虚拟对象。具体地，对象调整单元3031以所述环境场景的视频图像中的感兴趣区域的中心为虚拟对象的渲染位置，调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；对象渲染单元3032，根据所述虚拟对象的旋转矩阵和位移矩阵，在环境场景的视频图像的当前帧中的感兴趣区域内渲染所述虚拟对象。
本发明实施例的基于平面检测实现增强现实的装置，通过摄像机实时采集环境场景的视频图像，进行摄像机位姿跟踪并建立环境场景的三维地图，确定环境场景的视频图像中的用户感兴趣区域，通过检测用户感兴趣区域中的平面，进行虚拟对象的姿态调整，完成虚拟对象在环境场景的视频图像中的叠加渲染。本发明实施例基于稀疏点云检测环境场景中用户感兴趣区域中的平面，计算量小，运算效率高，适用于移动增强现实实现，同时能够使叠加渲染的虚拟对象的姿态与其叠加位置的平面保持一致，显著提升了虚拟对象与真实环境场景的贴合度，实现虚拟对象与真实环境无缝融合的效果。
实施例四
图7示出了本发明第四实施例的基于平面检测实现增强现实的方法的流程示意图。
本发明实施例的基于平面检测实现增强现实的方法，如图7所示，包括以下步骤：步骤S401，通过摄像机实时获取环境场景的视频图像；步骤S402，根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；步骤S403，提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合；步骤S404，根据所述地图点集合，检测所述感兴趣区域中的平面；步骤S405，根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
在上述技术方案中，所述步骤S401，还包括：根据所述环境场景的视频图像，确定所述环境场景的视频图像的当前帧对应的摄像机位姿；以及，根据所述环境场景的视频图像，构建所述环境场景的三维地图。具体地，实时采集环境场景的视频图像，通过逐帧提取环境场景的视频图像中的图像特征点，对环境场景的视频图像的相邻帧进行图像特征匹配，根据最小化重投影误差的方法计算当前帧对应的摄像机位于环境场景中的位置和方向信息，以及，根据环境场景的视频图像在三维空间中构建描述环境场景的空间三维地图，所述三维地图中包括地图点的位置和摄像机的位姿。
在上述技术方案，所述步骤S402，具体为：环境场景的视频图像实时采集过程中， 在屏幕中显示视频图像，根据用户在屏幕上进行触控操作的位置，确定以所述触控操作位置为中心的预设范围区域为所述环境场景的视频图像的当前帧中的感兴趣区域，其中，触控操作可以是点击、划动、圈选等，此处不作限定；或者，根据用户操控指令，摄像机对焦于环境场景中用户希望渲染虚拟对象的场景区域，所述场景区域包括平面或近似平面，例如桌面、墙面或地面等，确定所述场景区域为所述环境场景的视频图像的当前帧中的感兴趣区域。
在上述技术方案中，所述步骤S403，具体为：提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合FP。
在上述技术方案中，所述步骤S404，具体为：根据所述地图点集合FP中地图点的平均坐标值，将所述地图点集合FP中每一个地图点的坐标值减去所述平均坐标值，获取以所述环境场景的三维地图的坐标原点为中心的地图点集合MP；将所述地图点集合MP转换为数据矩阵并进行奇异值分解，确定以最小奇异值向量为法线的平面P；递归计算所述地图点集合MP中所有地图点到所述平面P的距离，判断距离小于预设阈值ε的地图点数量是否大于预设数量阈值l，若是，则确定所述平面P为所述感兴趣区域中的平面。
在上述技术方案中，所述步骤S405，具体为：调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；在所述环境场景的视频图像的当前帧中的感兴趣区域内，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。具体地，以所述环境场景的视频图像中的感兴趣区域的中心为虚拟对象的渲染位置，调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵，并据此渲染所述虚拟对象。
本发明实施例的基于平面检测实现增强现实的方法，通过摄像机实时采集环境场景的视频图像，进行摄像机位姿跟踪并建立环境场景的三维地图，确定环境场景的视频图像中的用户感兴趣区域，通过检测用户感兴趣区域中的平面，进行虚拟对象的姿态调整，完成虚拟对象在环境场景的视频图像中的叠加渲染。本发明实施例基于稀疏点云检测环境场景中用户感兴趣区域中的平面，运算效率高，适用于移动增强现实实现，同时能够使叠加渲染的虚拟对象的姿态与其叠加位置的平面保持一致，显著提升了虚拟对象与真实环境场景的贴合度，实现虚拟对象与真实环境无缝融合的效果。
实施例五
本实施例涉及基于平面检测实现增强现实的装置的结构示意图。由于本实施例的装置的结构示意图与图3至图6中所描述的装置相近，故在此处不再单独进行附图。
参照图3所示，主要包括：图像采集模块301：用于通过摄像机实时获取环境场景的视频图像；平面检测模块302：用于确定所述图像采集模块301获取的环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；对象渲染模块303：用于根据所述平面检测模块302检测到的所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
在该技术方案中，图像采集模块301通过摄像机实时获取环境场景的视频图像，基于同时定位与地图创建技术，确定环境场景的视频图像对应的摄像机位姿并构建环境场景的三维地图。平面检测模块302确定环境场景的视频图像中的用户感兴趣区域并检测感兴趣区域中的平面，对象渲染模块303根据感兴趣区域中的平面实现虚拟对象在环境场景的视频图像的当前帧中的叠加渲染。
在上述技术方案中，优选地，所述图像采集模块301，如图4所示，还包括：图像跟踪单元3011：用于根据所述环境场景的视频图像，确定所述环境场景的视频图像的当前帧对应的摄像机位姿；地图构建单元3012：用于根据所述环境场景的视频图像，构建所述环境场景的三维地图。
在该技术方案中，图像跟踪单元3011通过逐帧提取环境场景的视频图像中的图像特征点，对环境场景的视频图像的相邻帧进行图像特征匹配，通过最小化重投影误差的方法计算当前帧对应的摄像机位于环境场景中的位置和方向信息；地图构建单元3012根据环境场景的视频图像在三维空间中构建描述环境场景的空间三维地图，所述三维地图中包括地图点的位置和摄像机的位姿。
在上述技术方案中，优选地，平面检测模块302，如图5所示，包括：区域确定单元3021：用于根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；特征点提取单元3022：用于提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合；平面检测单元3023：用于根据所述地图点集合，检测所述感兴趣区域中的平面。
在该技术方案中，具体地，图像采集模块301进行环境场景的视频图像实时采集过程中，在屏幕中显示视频图像，所述区域确定单元3021根据用户在屏幕上进行触控操作的 位置，确定以所述触控操作位置为中心的预设范围区域为所述环境场景的视频图像的当前帧中的感兴趣区域，其中，触控操作可以是点击、划动、圈选等，此处不作限定；或者，根据用户操控指令，区域确定单元3021将摄像机对焦于环境场景中用户希望渲染虚拟对象的场景区域，所述场景区域包括平面或近似平面，例如桌面、墙面或地面等，确定所述场景区域为所述环境场景的视频图像的当前帧中的感兴趣区域。
在上述技术方案中，具体地，特征点提取单元3022，提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合FP。
在上述技术方案中，具体地，平面检测单元3023，根据所述地图点集合FP中地图点的平均坐标值，将所述地图点集合中每一个地图点的坐标值减去所述平均坐标值，获取以所述环境场景的三维地图的坐标原点为中心的地图点集合MP；将所述地图点集合MP转换为数据矩阵并进行奇异值分解，确定以最小奇异值向量为法线的平面P；递归计算所述地图点集合MP中所有地图点到所述平面P的距离，判断距离小于预设阈值ε的地图点数量是否大于预设数量阈值l，若是，则确定所述平面P为所述感兴趣区域中的平面。
在上述技术方案中，优选地，所述对象渲染模块303，如图6所示，包括：对象调整单元3031：用于调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；对象渲染单元3032：用于在所述环境场景的视频图像的当前帧中的感兴趣区域，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。具体地，对象调整单元3031以所述环境场景的视频图像中的感兴趣区域的中心为虚拟对象的渲染位置，调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；对象渲染单元3032，根据所述虚拟对象的旋转矩阵和位移矩阵，在环境场景的视频图像的当前帧中的感兴趣区域内渲染所述虚拟对象。
本发明实施例的基于平面检测实现增强现实的装置，通过摄像机实时采集环境场景的视频图像，进行摄像机位姿跟踪并建立环境场景的三维地图，确定环境场景的视频图像中的用户感兴趣区域，通过检测用户感兴趣区域中的平面，进行虚拟对象的姿态调整，完成虚拟对象在环境场景的视频图像中的叠加渲染。本发明实施例基于稀疏点云检测环境场景中用户感兴趣区域中的平面，计算量小，运算效率高，适用于移动增强现实实现，同时能够使叠加渲染的虚拟对象的姿态与其叠加位置的平面保持一致，显著提升了虚拟对象与真实环境场景的贴合度，实现虚拟对象与真实环境无缝融合的效果。
再次声明，本说明书中公开的所有特征，或公开的所有方法或过程中的步骤，除了互相排斥的特征和/或步骤以外，均可以以任何方式组合。
本说明书(包括任何附加权利要求、摘要和附图)中公开的任一特征，除非特别叙述，均可被其他等效或具有类似目的的替代特征加以替换。即，除非特别叙述，每个特征只是一系列等效或类似特征中的一个例子而已。
本发明并不局限于前述的具体实施方式。本发明可以扩展到任何在本说明书中披露的新特征或任何新的组合，以及披露的任一新的方法或过程的步骤或任何新的组合。

Claims (16)

  1. 一种基于平面检测实现增强现实的方法，其特征在于，包括以下步骤：
    通过摄像机实时获取环境场景的视频图像；
    确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；
    根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
  2. 根据权利要求1所述的基于平面检测实现增强现实的方法，其特征在于，所述通过摄像机实时获取环境场景的视频图像的步骤，还包括：
    确定所述环境场景的视频图像的当前帧对应的摄像机位姿；以及，
    根据所述环境场景的视频图像，构建所述环境场景的三维地图。
  3. 根据权利要求2所述的基于平面检测实现增强现实的方法，其特征在于，所述确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面的步骤，具体为：
    根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；或者，根据所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域；
    提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合；
    在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面。
  4. 根据权利要求3所述的基于平面检测实现增强现实的方法，其特征在于，所述在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面的步骤，具体为：
    在所述图像特征点集合中，通过随机抽样一致性算法选取预设数量的图像特征点；
    根据所述环境场景的三维地图，确定各个选取的图像特征点在所述环境场景的视频图像的参考帧中的对应图像点，所述参考帧为所述环境场景的视频图像的当前帧的前一关键帧或者前一相邻帧；
    根据所有选取的图像特征点及其在所述参考帧中的对应图像点，迭代计算单应性矩 阵，并根据所述单应性矩阵确定旋转矩阵R0和位移矩阵t0；
    根据当前帧图像与参考帧图像的旋转矩阵R和位移矩阵t，判断包括所述旋转矩阵R0和位移矩阵t0的矩阵与包括所述旋转矩阵R和位移矩阵t的矩阵之间的误差是否小于预设阈值ε，若是，则确定所有选取的图像特征点在同一平面上，并确定该平面为所述感兴趣区域中的平面。
  5. 根据权利要求4所述的基于平面检测实现增强现实的方法，其特征在于，所述确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面的步骤，还包括：
    根据所述图像特征点集合中到所述感兴趣区域中的平面的距离小于预设距离阈值的图像特征点，确定所述感兴趣区域中的平面的法线。
  6. 根据权利要求2所述的基于平面检测实现增强现实的方法，其特征在于，所述确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面的步骤，具体为：
    根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；
    提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合；
    根据所述地图点集合，检测所述感兴趣区域中的平面。
  7. 根据权利要求6所述的基于平面检测实现增强现实的方法，其特征在于，所述根据所述地图点集合，检测所述感兴趣区域中的平面的步骤，具体为：
    根据所述地图点集合中地图点的平均坐标值，获取以所述环境场景的三维地图的坐标原点为中心的地图点集合MP；
    将所述地图点集合MP转换为数据矩阵并进行奇异值分解，确定以最小奇异值对应的奇异向量为法线的平面P；
    递归计算所述地图点集合MP中所有地图点到所述平面P的距离，判断距离小于预设阈值ε的地图点数量是否大于预设数量阈值l，若是，则确定所述平面P为所述感兴趣区域中的平面。
  8. 根据权利要求5或7所述的基于平面检测实现增强现实的方法，其特征在于，所述根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象的步骤，具体为：
    调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；
    在所述环境场景的视频图像的当前帧中的感兴趣区域内，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。
  9. 一种基于平面检测实现增强现实的装置，其特征在于，包括：
    图像采集模块：用于通过摄像机实时获取环境场景的视频图像；
    平面检测模块：用于确定所述环境场景的视频图像的当前帧中的感兴趣区域，检测所述感兴趣区域中的平面；
    对象渲染模块：用于根据所述感兴趣区域中的平面，在所述环境场景的视频图像的当前帧中渲染虚拟对象。
  10. 根据权利要求9所述的基于平面检测实现增强现实的装置，其特征在于，所述图像采集模块，还包括：
    图像跟踪单元：用于确定所述环境场景的视频图像的当前帧对应的摄像机位姿；
    地图构建单元：用于根据所述环境场景的视频图像，构建所述环境场景的三维地图。
  11. 根据权利要求10所述的基于平面检测实现增强现实的装置，其特征在于，所述平面检测模块，包括：
    区域确定单元：用于根据用户操控指令或所述摄像机位姿，确定所述环境场景的视频图像的当前帧中的感兴趣区域；
    特征点提取单元：用于提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，得到图像特征点集合；
    平面检测单元：用于在所述图像特征点集合中，选取预设数量的图像特征点，并检测所述感兴趣区域中的平面。
  12. 根据权利要求11所述的基于平面检测实现增强现实的装置，其特征在于，所述平面检测单元，具体用于：
    在所述图像特征点集合中，通过随机抽样一致性算法选取预设数量的图像特征点；
    根据所述环境场景的三维地图，确定各个选取的图像特征点在所述环境场景的视频图像的参考帧中的对应图像点，所述参考帧为所述环境场景的视频图像的当前帧的前一关键帧或者前一相邻帧；
    根据所有选取的图像特征点及其在所述参考帧中的对应图像点，迭代计算单应性矩阵，并根据所述单应性矩阵确定旋转矩阵R0和位移矩阵t0；
    根据当前帧图像与参考帧图像的旋转矩阵R和位移矩阵t，判断包括所述旋转矩阵R0和位移矩阵t0的矩阵与包括所述旋转矩阵R和位移矩阵t的矩阵之间的误差是否小于预设阈值ε，若是，则确定所有选取的图像特征点在同一平面上，并确定该平面为所述感兴趣区域中的平面。
  13. 根据权利要求12所述的基于平面检测实现增强现实的装置，其特征在于，所述平面检测模块，还包括：
    法线确定单元：用于根据所述图像特征点集合中到所述感兴趣区域中的平面的距离小于预设距离阈值的图像特征点，确定所述感兴趣区域中的平面的法线。
  14. 根据权利要求10所述的基于平面检测实现增强现实的装置，其特征在于，所述平面检测模块，包括：
    区域确定单元：用于根据用户操控指令，确定所述环境场景的视频图像的当前帧中的感兴趣区域；
    特征点提取单元：用于提取所述环境场景的视频图像的当前帧中的感兴趣区域中的图像特征点，根据所述环境场景的三维地图，确定所述图像特征点对应的地图点，得到地图点集合；
    平面检测单元：用于根据所述地图点集合，检测所述感兴趣区域中的平面。
  15. 根据权利要求14所述的基于平面检测实现增强现实的装置，其特征在于，所述平面检测单元，具体用于：
    根据所述地图点集合中地图点的平均坐标值，获取以所述环境场景的三维地图的坐标原点为中心的地图点集合MP；
    将所述地图点集合MP转换为数据矩阵并进行奇异值分解，确定以最小奇异值对应的奇异向量为法线的平面P；
    递归计算所述地图点集合MP中所有地图点到所述平面P的距离，判断距离小于预设阈值ε的地图点数量是否大于预设数量阈值l，若是，则确定所述平面P为所述感兴趣区域中的平面。
  16. 根据权利要求13或15所述的基于平面检测实现增强现实的装置，其特征在于，所述对象渲染模块，包括：
    对象调整单元：用于调整所述虚拟对象的位置和/或角度，使所述虚拟对象的法线方向与所述感兴趣区域中的平面的法线方向一致，确定所述虚拟对象的旋转矩阵和位移矩阵；
    对象渲染单元：用于在所述环境场景的视频图像的当前帧中的感兴趣区域内，根据所述虚拟对象的旋转矩阵和位移矩阵渲染所述虚拟对象。
PCT/CN2017/094736 2016-07-29 2017-07-27 基于平面检测实现增强现实的方法及装置 WO2018019272A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201610617570.4A CN107665507B (zh) 2016-07-29 2016-07-29 基于平面检测实现增强现实的方法及装置
CN201610616770.8A CN107665505B (zh) 2016-07-29 2016-07-29 基于平面检测实现增强现实的方法及装置
CN201610616770.8 2016-07-29
CN201610617570.4 2016-07-29

Publications (1)

Publication Number Publication Date
WO2018019272A1 true WO2018019272A1 (zh) 2018-02-01

Family

ID=61015572

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/094736 WO2018019272A1 (zh) 2016-07-29 2017-07-27 基于平面检测实现增强现实的方法及装置

Country Status (1)

Country Link
WO (1) WO2018019272A1 (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108434742A (zh) * 2018-02-02 2018-08-24 网易（杭州）网络有限公司 游戏场景中虚拟资源的处理方法和装置
CN109636916A (zh) * 2018-07-17 2019-04-16 北京理工大学 一种动态标定的大范围虚拟现实漫游系统及方法
CN109685913A (zh) * 2018-12-21 2019-04-26 西安电子科技大学 基于计算机视觉定位的增强现实实现方法
CN110060343A (zh) * 2019-04-24 2019-07-26 百度在线网络技术（北京）有限公司 地图构建方法及系统、服务器、计算机可读介质
CN110223394A (zh) * 2019-05-16 2019-09-10 谷东科技有限公司 一种微弱光线条件下的ar显示方法
CN111311665A (zh) * 2020-03-12 2020-06-19 北京字节跳动网络技术有限公司 视频处理方法、装置及电子设备
CN111967342A (zh) * 2020-07-27 2020-11-20 杭州易现先进科技有限公司 平面参数设置的方法、装置、电子装置和存储介质
CN111986332A (zh) * 2020-08-28 2020-11-24 深圳市慧鲤科技有限公司 展示留言板的方法、装置、电子设备及存储介质
CN111986296A (zh) * 2020-08-20 2020-11-24 叠境数字科技（上海）有限公司 一种用于子弹时间的cg动画合成方法
CN114549766A (zh) * 2022-04-24 2022-05-27 成都纵横自动化技术股份有限公司 一种实时ar可视化方法、装置、设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103197980A (zh) * 2012-01-10 2013-07-10 华为终端有限公司 一种呈现增强现实技术内容的方法、装置及系统
CN103391411A (zh) * 2012-05-08 2013-11-13 索尼公司 图像处理设备、投影控制方法及程序
CN103810356A (zh) * 2012-11-05 2014-05-21 大连力天科技有限公司 基于增强现实技术及lbs功能的移动终端系统
CN104102678A (zh) * 2013-04-15 2014-10-15 腾讯科技（深圳）有限公司 增强现实的实现方法以及实现装置
CN106200956A (zh) * 2016-07-07 2016-12-07 北京时代拓灵科技有限公司 一种虚拟现实领域多媒体呈现和交互的方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103197980A (zh) * 2012-01-10 2013-07-10 华为终端有限公司 一种呈现增强现实技术内容的方法、装置及系统
CN103391411A (zh) * 2012-05-08 2013-11-13 索尼公司 图像处理设备、投影控制方法及程序
CN103810356A (zh) * 2012-11-05 2014-05-21 大连力天科技有限公司 基于增强现实技术及lbs功能的移动终端系统
CN104102678A (zh) * 2013-04-15 2014-10-15 腾讯科技（深圳）有限公司 增强现实的实现方法以及实现装置
CN106200956A (zh) * 2016-07-07 2016-12-07 北京时代拓灵科技有限公司 一种虚拟现实领域多媒体呈现和交互的方法

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11450061B2 (en) 2018-02-02 2022-09-20 Netease (Hangzhou) Network Co., Ltd. Processing method and apparatus for virtual resource in game scene
CN108434742A (zh) * 2018-02-02 2018-08-24 网易（杭州）网络有限公司 游戏场景中虚拟资源的处理方法和装置
CN109636916A (zh) * 2018-07-17 2019-04-16 北京理工大学 一种动态标定的大范围虚拟现实漫游系统及方法
CN109636916B (zh) * 2018-07-17 2022-12-02 北京理工大学 一种动态标定的大范围虚拟现实漫游系统及方法
CN109685913A (zh) * 2018-12-21 2019-04-26 西安电子科技大学 基于计算机视觉定位的增强现实实现方法
CN109685913B (zh) * 2018-12-21 2022-12-02 西安电子科技大学 基于计算机视觉定位的增强现实实现方法
CN110060343A (zh) * 2019-04-24 2019-07-26 百度在线网络技术（北京）有限公司 地图构建方法及系统、服务器、计算机可读介质
CN110060343B (zh) * 2019-04-24 2023-06-20 阿波罗智能技术(北京)有限公司 地图构建方法及系统、服务器、计算机可读介质
CN110223394A (zh) * 2019-05-16 2019-09-10 谷东科技有限公司 一种微弱光线条件下的ar显示方法
CN111311665A (zh) * 2020-03-12 2020-06-19 北京字节跳动网络技术有限公司 视频处理方法、装置及电子设备
CN111967342A (zh) * 2020-07-27 2020-11-20 杭州易现先进科技有限公司 平面参数设置的方法、装置、电子装置和存储介质
CN111967342B (zh) * 2020-07-27 2024-04-12 杭州易现先进科技有限公司 平面参数设置的方法、装置、电子装置和存储介质
CN111986296A (zh) * 2020-08-20 2020-11-24 叠境数字科技（上海）有限公司 一种用于子弹时间的cg动画合成方法
CN111986332A (zh) * 2020-08-28 2020-11-24 深圳市慧鲤科技有限公司 展示留言板的方法、装置、电子设备及存储介质
CN114549766A (zh) * 2022-04-24 2022-05-27 成都纵横自动化技术股份有限公司 一种实时ar可视化方法、装置、设备及存储介质

Similar Documents

Publication Publication Date Title
WO2018019272A1 (zh) 基于平面检测实现增强现实的方法及装置
CN107665506B (zh) 实现增强现实的方法及系统
CN107665505B (zh) 基于平面检测实现增强现实的方法及装置
TWI574223B (zh) 運用擴增實境技術之導航系統
CN107025662B (zh) 一种实现增强现实的方法、服务器、终端及系统
EP2915140B1 (en) Fast initialization for monocular visual slam
CN107665508B (zh) 实现增强现实的方法及系统
CN107665507B (zh) 基于平面检测实现增强现实的方法及装置
CN107025661B (zh) 一种实现增强现实的方法、服务器、终端及系统
JP2016526313A (ja) 総体的カメラ移動およびパノラマカメラ移動を使用した単眼視覚ｓｌａｍ
CN104881029B (zh) 基于一点ransac和fast算法的移动机器人导航方法
US11290705B2 (en) Rendering augmented reality with occlusion
Rodríguez et al. Obstacle avoidance system for assisting visually impaired people
JPWO2019021569A1 (ja) 情報処理装置、情報処理方法、及びプログラム
US11393179B2 (en) Rendering depth-based three-dimensional model with integrated image frames
TW202238449A (zh) 室內定位系統及室內定位方法
US20200211275A1 (en) Information processing device, information processing method, and recording medium
CN105339981B (zh) 用于使用一组基元配准数据的方法
WO2017003424A1 (en) Metric 3d stitching of rgb-d data
US20220068024A1 (en) Determining a three-dimensional representation of a scene
US11418716B2 (en) Spherical image based registration and self-localization for onsite and offsite viewing
Aliakbarpour et al. Geometric exploration of virtual planes in a fusion-based 3D data registration framework
CN112329723A (zh) 一种基于双目相机的多人人体3d骨骼关键点定位方法
CN112836544A (zh) 一种新型的坐姿检测方法
CN114529603B (zh) 一种基于激光slam与单目视觉slam融合的里程计方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17833575

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17833575

Country of ref document: EP

Kind code of ref document: A1