WO2018040982A1 - 一种用于增强现实的实时图像叠加方法及装置 - Google Patents

一种用于增强现实的实时图像叠加方法及装置 Download PDF

Info

Publication number
WO2018040982A1
WO2018040982A1 PCT/CN2017/098476 CN2017098476W WO2018040982A1 WO 2018040982 A1 WO2018040982 A1 WO 2018040982A1 CN 2017098476 W CN2017098476 W CN 2017098476W WO 2018040982 A1 WO2018040982 A1 WO 2018040982A1
Authority
WO
WIPO (PCT)
Prior art keywords
plane
rectangle
key frame
preset
current key
Prior art date
Application number
PCT/CN2017/098476
Other languages
English (en)
French (fr)
Inventor
黄敏
郑黎力
Original Assignee
成都理想境界科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 成都理想境界科技有限公司 filed Critical 成都理想境界科技有限公司
Publication of WO2018040982A1 publication Critical patent/WO2018040982A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20152—Watershed segmentation

Definitions

  • the present invention relates to the field of augmented reality, and in particular to a real-time image overlay method and apparatus for augmented reality.
  • Augmented Reality (English: Augmented Reality; abbreviation: AR) is one of the hot research topics in the field of machine vision in recent years.
  • the purpose of this technology is to superimpose the virtual image onto the image of the real environment through a special computer image and video processing algorithm in the real environment captured by the camera on the screen of the AR device, so that the user can obtain a new transcendence.
  • Realistic sensory experience When superimposing a virtual image onto an image of a real environment, it is necessary to obtain a digital model of the real environment in order to ensure the accuracy of the superimposition of the virtual image.
  • the point cloud data is first established by a laser scanner or a camera scanner, and the point cloud data is subjected to corresponding processing such as plane detection, and the digital processing of the real environment is completed before being provided to the AR technology.
  • An object of the present invention is to provide a real-time image superimposition method and apparatus for augmented reality, so as to solve the problem that the existing AR technology needs to be processed in advance for the digitization of the real environment, thereby increasing the user experience to the AR technology.
  • a first aspect of the embodiments of the present invention provides a real-time image overlay method for augmented reality, including:
  • the bonding surface of the main feature in the virtual image is pasted with the first plane, and the main feature is positive
  • the edge is disposed to be parallel or perpendicular to the edge of the first plane, and the forward edge is an intersection line between the preset orientation surface of the main feature and the preset bottom surface.
  • the determining the longest edge line on each of the planes includes:
  • the step of determining the longest edge line on the second plane in the current key frame is determined as follows to determine the longest edge line on each of the planes:
  • the preset condition is specifically: the length of the straight line is the longest and/or the difference between the sum of the number of data points in the preset range of the two sides of the straight line is the largest.
  • the method further comprises: determining a rectangle for each plane, the side of the rectangle being parallel to a line of the plane of the plane in which it is located;
  • the method further includes: adjusting the main feature according to a relative size of the matching surface and the rectangle of the third plane The size of the object.
  • the determining a rectangle for each plane includes:
  • the steps in the third plane in the current key frame are determined as follows to determine the rectangle in each of the planes:
  • the determining the seed point in the second planar image includes:
  • the first data point is used as the seed point.
  • the determining, according to the seed point, the rectangle in the second planar image includes:
  • the first rectangle is performed in the four directions of the seed point to the four sides of the first rectangle Expanded to obtain an enlarged second rectangle;
  • the largest area in the second rectangle corresponding to all the seed points is taken as the rectangle in the second planar image.
  • the first rectangle is enlarged according to the four directions of the seed point to the four sides of the first rectangle, and the enlarged second rectangle is obtained, which specifically includes:
  • first ratio after the re-detection is greater than the second threshold, then continue to expand in the same direction, otherwise stop expanding in the same direction and expand in other directions;
  • a second aspect of the embodiments of the present invention provides a real-time image superimposing apparatus for augmented reality, including:
  • a key frame determining unit configured to acquire video information when a monocular camera captures a preset scene in real time, and determine a key frame from the video information
  • a point cloud data establishing unit configured to establish point cloud data in real time according to a current key frame and a preceding key frame of the current key frame, where a parallax angle between the current key frame and the preceding key frame is greater than a preset parallax An angle, each of the point cloud data includes three-dimensional coordinates;
  • a plane determining unit configured to project point cloud data corresponding to the current key frame onto the current key frame, and determine, in the current key frame, according to a projection result and a super pixel segmentation result of the current key frame flat;
  • An edge straight line determining unit configured to determine a longest edge line on each of the planes according to point cloud data corresponding to each of the current key frames
  • An image superimposing unit configured to: when the virtual image is superimposed on the first plane in the current key frame, fit the bonding surface of the main feature in the virtual image to the first plane, and
  • the positive side of the main feature is disposed to be parallel or perpendicular to the edge of the first plane, and the forward side is an intersection line between the preset orientation surface of the main feature and the preset bottom surface.
  • the edge line determining unit is specifically configured to determine the longest edge line on each of the planes by determining the longest edge line on the second plane in the current key frame:
  • the preset condition is specifically: the length of the straight line is the longest and/or the difference between the sum of the number of data points in the preset range of the two sides of the straight line is the largest.
  • the real-time image superimposing apparatus further includes a rectangle determining unit, configured to determine one for each plane when the edge straight line determining unit determines the longest edge straight line on each of the planes a rectangle whose one side is parallel to the edge of the plane in which it is located;
  • the image superimposing unit is further configured to adjust the main size according to a relative size of a rectangle of the bonding surface and the first plane when superimposing a virtual image on a first plane in the current key frame The size of the feature.
  • the rectangle determining unit is specifically configured to determine a rectangle in each of the planes by determining a rectangle in a third plane in the current key frame as follows:
  • the rectangle determining unit is specifically configured to determine a seed point in the second planar image by calculating data of a preset size region centered on each data point in the second planar image. Point proportion, if the data point of the first data point is the highest, the first data point is taken as the seed point.
  • the rectangle determining unit is specifically configured to determine, according to the seed point, a rectangle in the second planar image by: setting a first centering on the seed point and an area equal to a first threshold a rectangle, and detecting a common area between the first rectangle and the third plane, if the first ratio between the common area and the area of the first rectangle is greater than a second threshold, then the seed is Pointing to the four directions of the four sides of the first rectangle to enlarge the first rectangle, obtaining an enlarged second rectangle, and using the largest area in the second rectangle corresponding to all the seeds as the second A rectangle in a flat image.
  • the rectangle determining unit is specifically configured to obtain the enlarged second rectangle by expanding the direction of the seed point to the four sides of the first rectangle, and expanding in each direction. After the first ratio is re-detected, if the first ratio after the re-detection is greater than the second threshold, the expansion is continued in the same direction, otherwise the expansion is stopped in the same direction and expanded in other directions. Until the enlargement is stopped in each direction, the finally obtained rectangle is taken as the second rectangle.
  • the plane in the current key frame and the longest edge line on each plane are determined, and the virtual image is superimposed to the first in the current key frame.
  • the bonding surface of the main feature in the virtual image is attached to the first plane, and the positive side of the main feature is set to
  • the technical solution of the edge of the first plane is parallel or perpendicular, so that the visual experience of the AR technology can be provided to the user without digitizing the real environment in advance, thereby reducing the user experience to the pre-step of the AR technology, and the real-time performance is strong.
  • the scope of application is also greatly improved, and the actual edge of the plane in the real environment is also combined, which significantly improves the authenticity of the main features in the virtual image, thereby reducing or avoiding the possibility that the user perceives that the main feature is virtual. It is then able to provide an immersive experience to the user.
  • the technical solution of adjusting the size of the main feature is such that the superimposed virtual
  • the authenticity of the main features in the image is further enhanced, thereby further reducing or avoiding the possibility that the user perceives that the primary feature is virtual, and in turn is better able to provide an immersive experience to the user.
  • FIG. 1 is a schematic flowchart of a real-time image superposition method for augmented reality according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a monocular camera capturing a preset scene in real time according to an embodiment of the present invention
  • FIG. 3 is a schematic block diagram of a real-time image superimposing apparatus for augmented reality according to an embodiment of the present invention.
  • Embodiments of the present invention provide a real-time image overlay method for augmented reality. Please refer to FIG. 1.
  • FIG. 1 is a schematic flowchart diagram of a real-time image overlay method for augmented reality according to an embodiment of the present invention. As shown in FIG. 1, the method mainly includes steps S1 to S5.
  • step S1 video information when a monocular camera captures a preset scene in real time is acquired, and a key frame is determined from the video information.
  • the monocular camera can be disposed, for example, on a head mounted display device or a glasses type display device, and is not limited herein.
  • the corresponding video information can be obtained.
  • a key strategy can be used to determine key frames in the video information. For example, a new key frame can be determined when the interval between the key frame and the key frame is too far, or The key frames in the video information can be determined by a pixel frame averaging method, a histogram frame averaging method, or a mixture of both.
  • the criterion chosen is to prioritize the dissimilarity between key frames, that is, based on the similarity between frames, each time finding key frames, the minimum similarity between key frames is guaranteed to make the key Frames have the largest amount of information and will not be described here.
  • step S2 point cloud data is established in real time according to the current key frame and the previous key frame of the current key frame, and the parallax angle between the current key frame and the previous key frame is greater than the preset parallax angle, and each of the point cloud data Data points include three-dimensional coordinates.
  • the parallax angle between the current key frame and the preceding key frame is greater than the preset parallax angle
  • the parallax angle can be obtained by stereo matching between the current key frame and the preceding key frame, and then according to the triangulation
  • the principle is that the three-dimensional information of each point in the point cloud data corresponding to the current key frame can be restored, that is, each data point in the point cloud data includes three-dimensional coordinates.
  • the preset parallax angle is small, it is susceptible to noise, thereby affecting the accuracy of the three-dimensional coordinates of each data point in the point cloud data; if the preset parallax angle is large, the matching is difficult, thereby Unable to establish point cloud data. Therefore, the specific value of the preset parallax angle can be set by the person skilled in the art according to the actual situation to meet the actual situation, and will not be described here.
  • step S3 the point cloud data corresponding to the current key frame is projected onto the current key frame, and the plane in the current key frame is determined according to the projection result and the super pixel segmentation result of the current key frame.
  • the super key segmentation is performed on the current key frame to obtain a super pixel segmentation result
  • the point cloud data obtained in step S2 is projected onto the current key frame to obtain a projection result.
  • there is a projection point on each super pixel on the current key frame so that the plane corresponding to each super pixel can be calculated by the projection point associated with each super pixel, so that the plane in the current key frame can be determined.
  • the virtual image needs to be superimposed on the current key frame, it can be selected to be specifically superimposed on a certain plane on the current key frame according to the calculated plane, and the selection manner is not introduced here.
  • step S4 the longest edge line on each plane is determined according to the point cloud data corresponding to each plane in the current key frame.
  • each plane in the current key frame corresponds to a part of the point cloud data, so that the point cloud data corresponding to each plane can be used.
  • the longest edge line on each plane is determined by algorithms such as edge detection and line detection.
  • step S5 when the virtual image is superimposed on the first plane in the current key frame, the bonding surface of the main feature in the virtual image is pasted with the first plane, and the positive side of the main feature is set.
  • Straight to the edge of the first plane Lines are parallel or vertical.
  • the positive side is the intersection line between the preset orientation face of the main feature and the preset bottom face.
  • the AR experience can generally be provided to users in two ways.
  • the first is to record the external environment through the camera, superimpose the virtual image on the recorded image information, and then transfer the superimposed image to the human eye.
  • the second method is to transmit the light of the preset scene to the human eye through the lens, and simultaneously project the light of the virtual image onto the lens, so that the light of the virtual image is merged into the light of the preset scene and then transmitted to the human eye.
  • the main feature in the virtual image is a virtual doll as an example.
  • the matching surface of the doll can be set as the bottom surface of the doll, and the forward side of the doll is also set to be parallel with the edge of the desktop. Or vertical, the positive edge is the intersection of the front and bottom of the doll.
  • the bottom surface of the doll will fit on the table top, and the forward side of the doll will be parallel or perpendicular to the edge of the table.
  • the doll looks like a real doll is set on the tabletop of the table, and the front of the doll faces the actual edge of the tabletop or perpendicular to the actual edge of the tabletop, and the user sees the doll through the augmented reality device, Immediately aware that the doll is virtual, the doll's authenticity has been significantly improved, thereby reducing or avoiding the possibility that the user perceives that the main feature is virtual.
  • the monocular camera is used to capture the preset scene in real time, the plane in the current key frame and the longest edge line on each plane are determined, and the virtual image is superimposed on the current key frame.
  • the bonding surface of the main feature in the virtual image is adhered to the first plane, and the positive side of the main feature is set to be parallel or perpendicular to the edge of the first plane, so that it is not necessary
  • Digital processing of the real environment in advance can provide the user with the visual experience of the AR technology, reducing the user's experience of the AR technology's pre-steps, real-time performance, and the scope of application is greatly improved, and also combines the real environment in the plane.
  • the actual edge significantly improves the authenticity of the main features in the virtual image, thereby reducing or avoiding the possibility of the user perceiving that the primary feature is virtual, and in turn providing an immersive experience to the user.
  • step S1 video information when a monocular camera captures a preset scene in real time is acquired, and a key frame is determined from the video information.
  • FIG. 2 is a schematic diagram of a monocular camera capturing a preset scene in real time according to an embodiment of the present invention.
  • the object 201 may be a table, a wall or a ground, or the like.
  • the monocular camera 202 captures the object 201 in the preset scene in real time during the movement and rotation.
  • the image frame obtained by the monocular camera 202 at point A1 is F1
  • the image frame obtained at point A2 is F2
  • F1 and F2 are set as key frames extracted from the video information.
  • F2 is the current key frame
  • F1 is the previous key frame of F2.
  • F1 and F2 may or may not be adjacent, and no limitation is imposed herein.
  • the two steps of the monocular camera capturing the preset scene in real time and determining the key frame from the video information are performed simultaneously, and the video information is not determined after the video shooting is completed. Key frame this step.
  • step S1 After the key frame of the video information is determined in step S1, the real-time image superimposing method provided by the embodiment of the present invention proceeds to step S2: real-time establishing point cloud data, current key frame and current point according to the current key frame and the current key frame of the current key frame.
  • the parallax angle between the previous key frames is greater than the preset parallax angle, and each data point in the point cloud data includes three-dimensional coordinates.
  • the step S2 may include: obtaining a parallax angle between the current key frame and the preceding key frame by performing stereo matching on the current key frame and the preceding key frame; and establishing point cloud data in real time according to the triangulation principle.
  • the monocular camera 202 is in the key frame F1 shot at the A1 point, and the actual point P12 is the point P1, and the point is taken at the A2 point.
  • the actual point P12 corresponds to the point P2.
  • the parallax angles of the key frame F1 and the key frame F2 can be obtained, that is, the triangle formed by the point P1, the point P2, and the point P12 can be obtained.
  • the triangle formed by the point P1, the point P2 and the point P12 is solved, and the coordinates of the point P12 can be obtained.
  • the points on the object 201 that are simultaneously imaged and matched on the key frame F1 and the key frame F2 in the real environment can obtain their coordinates, thus, it is possible to establish inclusion.
  • Point cloud data for all feature points on object 201 in a real environment can obtain all feature points on object 201 in a real environment.
  • the point cloud data obtained here is obtained according to the key frame F1 and the key frame F2, and is a semi-dense (English: Semi-Dense) point cloud data.
  • the density of data points in semi-dense point cloud data is between sparse point cloud data and dense point cloud data.
  • the coordinates of each point in the point cloud data are relative coordinates, but each point The relative position between them is fixed.
  • point cloud data such as ORB
  • SLAM monocular concurrent localization and mapping
  • step S2 After obtaining the point cloud data in step S2, the real-time image superimposing method provided by the embodiment of the present invention proceeds to step S3, that is, the point cloud data corresponding to the current key frame is projected onto the current key frame, and according to the projection result and the current key frame. The result of the superpixel segmentation determines the plane in the current keyframe.
  • the step S3 may include: performing a super pixel segmentation process on the current key frame to obtain a super pixel set of the current key frame; projecting the point cloud data corresponding to the current key frame onto the current key frame, and obtaining the data point one by one.
  • Corresponding projection point traverse all projection points, detecting the shortest distance from each projection point to each super pixel, and first when the shortest distance from the first projection point to the first super pixel is less than the first preset value
  • the first data point corresponding to the projection point is associated with the first super pixel; according to the data point associated with each super pixel, a plane equation corresponding to each super pixel is obtained, thereby obtaining a plane equation corresponding to all super pixels in the current key frame.
  • superpixel refers to irregular pixel blocks with certain visual meanings composed of adjacent pixels with similar features such as grayscale, texture, color or brightness, and superpixel segmentation is one.
  • the picture is a pixel-level (English: pixel-level) picture, divided into regional-level (English: district-level) pictures, each feature of each sub-area has a strong consistency, for example, the gray level of each sub-area is very Close.
  • the superpixel segmentation algorithm is also called the generation algorithm.
  • superpixel segmentation algorithms There are many kinds of superpixel segmentation algorithms, which can be divided into two types based on graph theory and gradient descent method.
  • the superpixel segmentation algorithm based on the graph theory may include, for example, a graph-based image segmentation algorithm, a superpixel lattice algorithm, and the like.
  • the super pixel segmentation algorithm based on the gradient descent method may include, for example, a watershed algorithm (English: watershed), a mean shift algorithm (English: mean-shift) or a simple linear iterative cluster (English: simple linear iterative cluster; SLIC). Wait, I won't go into details here.
  • each super pixel in the super pixel set includes one or more pixels. Since each superpixel represents a region with low texture, low gradient values, and similar colors, it can be approximated that each superpixel is a planar region.
  • the point cloud data corresponding to the current key frame is projected onto the current key frame, so that each data point in the point cloud data can correspond to one projection point, and each projection point is on the plane of the current key frame.
  • the two-dimensional point, the specific process of projection will not be described here.
  • the first projection point can be calculated The shortest distance from each super pixel.
  • the specific calculation method may be as follows: the first projection point and the centroid of all the super pixels are respectively connected by a connecting line, and the distance between the first projection point and the intersection of the connecting line and the edge of each super pixel is the shortest distance.
  • the point cloud may be The data point corresponding to the first projection point in the data is associated with the first super pixel, and so on, until all the data points in the point cloud data corresponding to the current key frame are associated with the super pixel in the current key frame.
  • each data point associated with the super pixel in order to ensure the accuracy of the obtained plane equation corresponding to the super pixel, it is necessary to first process each data point associated with the super pixel.
  • the outer point in each super pixel associated data point can be removed by a random sampling consistency algorithm (English: RANdom Sample Consensus; abbreviation: RANSAC).
  • RANSAC Random Sample Consensus
  • a certain number of sample data points can be randomly extracted therefrom, for example, 5 or 10 sample data points can be extracted therefrom.
  • the number of sample data points extracted is set to m.
  • the sample matrix is set to A m*3 .
  • V T is a 3*3 square matrix called singular vector.
  • the singular vector corresponding to the smallest singular value is the corresponding corresponding to the first super pixel.
  • the normal vector of the plane equation can be obtained by substituting any of the above sample data points to obtain the plane equation corresponding to the first superpixel, which will not be described here.
  • the centroid coordinates of the extracted sample data points can be calculated first, and then substituted into the normal vector of the plane corresponding to the first super pixel, thereby obtaining a more accurate plane equation, and thus no Let me repeat.
  • the distance of the data point associated with the first superpixel to the plane corresponding to the first plane equation is calculated. For example, taking a point P j (X j , Y j , Z j ) in the data point associated with the first super pixel as an example, the distance from P j to the plane corresponding to the first plane equation is:
  • a data point whose distance between the planes corresponding to the first plane equation is smaller than the first set value is set as an inner point, and a distance between a plane corresponding to the first plane equation is greater than or equal to the data point of the first set value
  • Set the outer point so that the excluded point can be removed from the data point associated with the first super pixel, and then the plane equation corresponding to the first super pixel is recalculated according to the data point associated with the first super pixel of the excluded point.
  • the specific calculation process is as before, and will not be described here. Set the plane equation obtained in this calculation to the second plane equation. Since the outer point is eliminated when calculating the second plane equation, the accuracy of the plane equation obtained by the calculation can be effectively improved, and will not be described here.
  • the distance between the data point associated with the first super pixel and the second plane equation can be calculated again, and the data point whose distance from the plane equation is smaller than the first set value is set as an inner point, and the sub data is counted. Concentrating the number of inner points belonging to the second fitting plane, and calculating the average distance of all the data points in the sub-data set to the plane equation is For use in subsequent steps.
  • a plurality of second plane equations can be obtained by cyclically performing the steps of calculating the first plane equation by extracting sample data points from the data points associated with the first superpixel and calculating the second plane equation again. Since the sample data points randomly selected each time are inconsistent, there will be some difference in the second plane equation finally obtained in each cycle. Generally speaking, the number of inner points of the second plane equation obtained in each cycle is different. Therefore, the number of inner points in the N second plane equations can be determined as the plane corresponding to the first super pixel. The equation, ie the plane equation, can be applied to the greatest extent to all data points associated with the first superpixel.
  • the plane equation corresponding to each super pixel in the current key frame can be calculated, so that the plane in the current key frame can be determined.
  • the real-time image superposition method provided by the embodiment of the present invention further includes:
  • the edge of the first super pixel can be obtained, so that the first The edge of the superpixel is projected onto an N frame keyframe adjacent to the current keyframe.
  • N 2
  • the edge of the first super pixel may be projected onto the previous key frame and the next key frame of the current key frame, or the edge of the first super pixel may be projected to the current key frame.
  • the edge of the first super pixel may be projected onto the first two frame key frames and the last two frame key frames of the current key frame, or the edge of the first super pixel may be projected to the front of the current key frame.
  • the edge of the first super-pixel can also be projected onto the previous key frame and the last three-frame key frame of the current key frame, which is not limited herein.
  • Those skilled in the art can select a suitable key frame for projection according to actual conditions to meet the needs of the actual situation, and will not be described herein.
  • the projection error of the edge of the matched super pixel in each key frame is calculated, which can be calculated by the following expression:
  • F(T j , p u ) is the projection point corresponding to P u in the jth key frame
  • the second preset value may be, for example, 1 pixel or 3 pixels or the like.
  • a person skilled in the art can set a specific value of the second preset value according to the actual situation to meet the needs of the actual situation, and details are not described herein again.
  • the plane equation corresponding to the first super pixel has a match, such that the plane equation corresponding to the first super pixel has more than the first preset number.
  • the plane equation corresponding to the first super pixel is the correct equation, that is, the plane equation of the super pixel in the current key frame can be applied to the super pixel matching the adjacent key frame, that is, indicating the current key
  • the plane equation of the superpixel in the frame is the correct equation.
  • step S3 After determining the plane in the current key frame by the step S3, the real-time image superimposing method provided by the embodiment of the present invention proceeds to step S4, that is, determining the longest length on each plane according to the point cloud data corresponding to each plane in the current key frame.
  • the edge of the line is straight.
  • Step S4 may specifically include step S41. Go to step S44.
  • step S41 the point cloud data corresponding to the second plane is converted into two-dimensional coordinates by being projected onto the first preset plane to obtain a first plane image, and the first preset plane is parallel to the second plane.
  • the first preset plane may be determined first, and the first preset plane is parallel to the second plane.
  • the first preset plane can be easily determined, and details are not described herein again. Then, all the data points in the point cloud data corresponding to the second plane are projected onto the first preset plane, so that the three-dimensional coordinates of the data points in the point cloud data corresponding to the second plane are converted into two-dimensional coordinates. Thereby a first planar image is obtained.
  • step S42 the first planar image is subjected to morphological processing, and the edge image of the first planar image is determined according to the processing result.
  • the first planar image may be subjected to expansion etching or the like to remove burrs in the two-dimensional image, and then edge detection may be performed on the processed first planar image.
  • edge detection may be performed on the processed first planar image.
  • the multi-directional morphological structural elements having multiple directions can be applied to obtain the edge detection results in multiple directions by applying basic morphology operations, and then the results are normalized and weighted and summed to obtain the final image edge.
  • the selection of the number of directions can be set by the person skilled in the art according to the actual situation to meet the needs of the actual situation, and will not be described here.
  • step S43 edge image detection is performed on the edge image, and a straight line is selected according to a preset condition.
  • the edge image may be passed through a Hough Transform (English: Hough Transform) algorithm, a Freeman (person name) criterion based line detection algorithm or a ruler creep algorithm (English: cankerworm crawlalgorithm), etc., from the edge image Determine the straight line. Since there are a plurality of edge lines in the edge image, it is necessary to select a line according to a preset condition.
  • Hough Transform English: Hough Transform
  • Freeman person name
  • ruler creep algorithm English: cankerworm crawlalgorithm
  • the preset condition may be that the length of the straight line is the longest and/or the difference between the sum of the number of data points in the preset range on both sides of the straight line is the largest.
  • the longest line length means that the line is the longest side in the edge image, that is, the line is the most prominent feature of the plane corresponding to the edge image.
  • the difference between the sum of the number of data points in the preset range on both sides of the line is the largest” means that there are relatively more data points on one side and relatively fewer data points on the other side of the line.
  • the line that satisfies this condition is also The most salient feature of the plane corresponding to the edge image.
  • the straight line represents the actual edge of the plane in the real environment.
  • step S44 it is determined that the straight line satisfying the preset condition is the longest edge line on the second plane, and will satisfy The straight line of the preset condition is projected into the second plane in a reverse projection manner.
  • the straight line satisfying the preset condition is an edge straight line, and this can be backprojected by means of back projection.
  • the edge line is inversely projected onto the point cloud data corresponding to the second plane, so that the actual edge line of the second plane can be determined.
  • steps S41-S44 are repeatedly performed for each of the current key frames, that is, the longest edge line on each plane can be determined, which will not be described in detail herein.
  • step S4 the real-time image superposition method provided by the embodiment of the present invention proceeds to step S5, that is, when the virtual image is superimposed on the first plane in the current key frame, The bonding surface of the main feature in the virtual image is adhered to the first plane, and the positive side of the main feature is set to be parallel or perpendicular to the edge of the first plane, and the positive side is the preset orientation of the main feature.
  • step S5 that is, when the virtual image is superimposed on the first plane in the current key frame,
  • the bonding surface of the main feature in the virtual image is adhered to the first plane, and the positive side of the main feature is set to be parallel or perpendicular to the edge of the first plane, and the positive side is the preset orientation of the main feature.
  • the intersection line between the face and the preset bottom face is the intersection line between the face and the preset bottom face.
  • the matching surface of the doll can be set. It is the bottom surface of the doll, and the forward side of the doll is also set to be parallel or perpendicular to the edge of the table top. The positive side is the intersection line between the front and the bottom of the doll, so that a virtual doll is superimposed.
  • the bottom of the doll After the side of the table, the bottom of the doll will fit on the table, and the front side of the doll will be parallel or perpendicular to the edge of the table, so that the doll looks like a real doll is set up On the table top of the table, and the front side of the doll faces the actual edge of the desktop or perpendicular to the actual edge of the desktop.
  • the doll When the user sees the doll through the augmented reality device, the doll is not immediately perceived as virtual, and the authenticity of the doll has A significant increase is made to reduce or avoid the possibility that the user perceives that the primary feature is virtual.
  • the monocular camera is used to capture the preset scene in real time, the plane in the current key frame and the longest edge line on each plane are determined, and the virtual image is superimposed on the current key frame.
  • the bonding surface of the main feature in the virtual image is adhered to the first plane, and the positive side of the main feature is set to be parallel or perpendicular to the edge of the first plane, so that it is not necessary
  • Digital processing of the real environment in advance can provide the user with the visual experience of the AR technology, reducing the user's experience of the AR technology's pre-steps, real-time performance, and the scope of application is greatly improved, and also combines the real environment in the plane.
  • the actual edge significantly improves the authenticity of the main features in the virtual image, thereby reducing or avoiding the possibility of the user perceiving that the primary feature is virtual, and in turn providing an immersive experience to the user.
  • the real-time image superimposing method provided by the embodiment of the present invention further includes: determining a rectangle for each plane, one side of the rectangle, when determining the longest edge line on each plane by step S4 It is parallel to the edge of the plane in which it is located.
  • the real-time image superimposing method provided by the embodiment of the present invention further includes: adjusting the main feature according to the relative size of the bonding surface and the rectangle when the bonding surface of the main feature in the virtual image is attached to the first plane. size.
  • the relative size between the bonding surface of the main feature in the virtual image and the rectangle in the plane to be superimposed can be ensured, and the size of the main feature in the virtual image can be adjusted, so that the size of the main feature can be adapted to The size of the plane to be superimposed avoids giving the user an unreal feeling.
  • determining a rectangle for each plane may specifically include: determining the rectangles in each plane by the steps S61-S64 of determining the rectangles in the third plane in the current key frame.
  • step S61 the point cloud data corresponding to the third plane is converted into two-dimensional coordinates by being projected onto the first preset plane to obtain a second planar image, and the first preset plane is parallel to the third plane.
  • the projection process here is consistent with the process of projecting the point cloud data corresponding to the second plane to the first preset plane described in the foregoing section, and details are not described herein again.
  • step S62 a seed point in the second planar image is determined, and the seed point is a center point having the highest proportion of data points in the preset size region in the second planar image.
  • each data point in the second planar image may be traversed, and the data point proportion of the preset size area centered on each data point is calculated. For example, taking an area with a preset size of 10*10 as an example, there are 100 data points in a preset size area centered on the first data point, and a preset size area centered on the second data point. There are 85 data points in the data, and there are 25 data points in the preset size area centered on the third data point, and the first data point can be used as the seed point.
  • the specific value of the preset size area may be set by a person skilled in the art to meet the actual situation, and is not limited herein.
  • the preset area may continue to be expanded until the number of data points becomes a preset number, for example, 5 or 10 and many more.
  • the seed point selected by the step S62 is not at the edge portion of the plane, thereby ensuring the correctness of the rectangle obtained in the subsequent portion.
  • step S63 a rectangle in the second planar image is determined based on the seed point.
  • the first rectangle may be set first with the seed point as the center, and the area of the first rectangle is equal to the first threshold.
  • the first threshold may be set by a person skilled in the art according to actual conditions. If the actual area of the plane is large, the first threshold may be larger, and if the plane is smaller, the first threshold may be smaller. For example, the first threshold may be 10*12. Next, a common area between the first rectangle and the third plane is detected. Then, it is judged whether the first ratio between the common area and the area of the first rectangle is greater than the second threshold.
  • the second threshold is also set by a person skilled in the art according to actual conditions. For example, the second threshold may be 80%, 90%, or the like. In this embodiment, if the first ratio is greater than the second threshold, the first rectangle may be expanded in four directions from the seed point to the four sides of the first rectangle.
  • setting the second threshold to less than 100% is to obtain the largest rectangle in the plane, and ensuring that the main features in the virtual image can be presented to the user in the largest area or volume, thereby being able to display to the user. More details.
  • the first rectangle may be enlarged according to the following manner: the direction of the seed point to the four sides of the first rectangle is respectively expanded; after each direction is expanded, the first ratio is re-detected; if re-detected If the first ratio is greater than the second threshold, then the expansion continues in the corresponding direction, otherwise (ie, the first ratio is less than or equal to the second threshold) stops expanding in the corresponding direction and expands in other directions; until each The direction is stopped expanding, and the finally obtained rectangle is taken as the second rectangle.
  • the second rectangle corresponding to each seed point can be obtained, and the second rectangle having the largest area is used as the rectangle in the second plane image.
  • step S64 the endpoint coordinates of the rectangle in the second planar image are converted into three-dimensional coordinates in a reverse projection manner, and a rectangle is determined for the third plane.
  • the four end point coordinates of the rectangle are respectively processed according to the inverse process of the projection in step S61, thereby obtaining four end points having three-dimensional coordinates, thereby determining a rectangle for the third plane, that is, the side of the table.
  • the image superimposing method provided by the embodiment of the present invention further includes: adjusting the main feature proportionally according to the relative size of the bonding surface and the rectangle. the size of.
  • the bonding surface of the main feature in the virtual image also has a certain size. If the bonding surface is smaller than the rectangle, the main feature can be attached to the rectangle of the plane according to the original size of the virtual image.
  • the original size of the virtual image can be increased by a preset ratio, so that the main feature is larger, and the bonding surface of the main feature is not larger than the rectangle of the plane; if the bonding surface is larger than the rectangle, the virtual image can be followed. Scale down so that the bonding surface of the main features is smaller than or equal to the rectangle.
  • the ratio you need to set a minimum value, otherwise it will affect the display of the main features, if the matching surface of the main features is the smallest. After the value is smaller and smaller, it is still larger than or far larger than the rectangular surface of the plane. You can select other planes for image overlay, which will not be described here.
  • the bonding surface of the main feature in the virtual image is attached to the plane rectangle, it is not attached.
  • the surface is larger or larger than the plane so that the main features appear to be suspended in the air, so that the authenticity of the main features in the superimposed virtual image is further improved, thereby further reducing or avoiding the user perceiving that the main feature is virtual.
  • the possibilities are better able to provide users with an immersive experience.
  • FIG. 3 is a schematic block diagram of a real-time image superimposing apparatus for augmented reality according to an embodiment of the present invention.
  • the real-time image superimposing apparatus mainly includes a key frame determining unit 301, a point cloud data establishing unit 302, a plane determining unit 303, an edge straight line determining unit 304, and an image superimposing unit 305.
  • the key frame determining unit 301 is configured to acquire video information when a monocular camera captures a preset scene in real time, and determine a key frame from the video information.
  • the point cloud data establishing unit 302 is configured to establish point cloud data in real time according to the current key frame and the preceding key frame of the current key frame, where the parallax angle between the current key frame and the preceding key frame is greater than a preset
  • the parallax angle, each of the point cloud data includes three-dimensional coordinates.
  • the plane determining unit 303 is configured to project the point cloud data corresponding to the current key frame onto the current key frame, and determine the current key frame according to the projection result and the super pixel segmentation result of the current key frame. The plane.
  • the edge line determining unit 304 is configured to determine a longest edge line on each of the planes according to the point cloud data corresponding to each of the current key frames.
  • the image superimposing unit 305 is configured to, when the virtual image is superimposed on the first plane in the current key frame, fit the bonding surface of the main feature in the virtual image to the first plane, and
  • the positive side of the main feature is disposed to be parallel or perpendicular to an edge of the first plane, and the forward side is an intersection line between the preset orientation surface of the main feature and the preset bottom surface.
  • the edge line determining unit 304 is specifically configured to determine the longest edge line on each plane in the following step of determining the longest edge line on the second plane in the current key frame:
  • the preset condition is specifically: the length of the straight line is the longest and/or the difference between the sum of the number of data points in the preset range of the two sides of the straight line is the largest.
  • the real-time image superimposing apparatus further includes a rectangle determining unit 306, and when the edge straight line determining unit 304 determines the longest edge straight line on each of the planes, Each plane defines a rectangle whose one side is parallel to the edge of the plane in which it is located.
  • the image superimposing unit 305 is further configured to: when the bonding surface of the main feature in the virtual image is pasted with the third plane, according to the rectangular shape of the bonding surface and the third plane The relative size is adjusted to adjust the size of the main feature.
  • the rectangle determining unit 306 is specifically configured to determine a rectangle in each plane in the following step of determining a rectangle in a third plane in the current key frame:
  • the rectangle determining unit 306 is specifically configured to determine a seed point in the second plane image by calculating a preset size area centered on each data point in the second plane image. The data point is proportioned, and if the data point of the first data point is the highest, the first data point is used as the seed point.
  • the rectangle determining unit 306 is specifically configured to determine, according to the seed point, a rectangle in the second planar image by setting a step that is centered on the seed point and has an area equal to the first threshold. a rectangle, and detecting a common area between the first rectangle and the third plane, if the first ratio between the common area and the area of the first rectangle is greater than a second threshold, then Seed points to the four directions of the four sides of the first rectangle The first rectangle is enlarged to obtain an enlarged second rectangle, and the largest area in the second rectangle corresponding to all the seed points is taken as a rectangle in the second plane image.
  • the rectangle determining unit 306 is specifically configured to obtain an enlarged second rectangle by expanding the direction of the seed point to the four sides of the first rectangle, and pressing each Re-detecting the first ratio after the direction is expanded, and continuing to expand in the same direction if the first ratio after re-detection is greater than the second threshold, otherwise (ie, if the first ratio is less than or equal to the The second threshold) stops expanding in the same direction and expands in the other directions until the enlargement is stopped in each direction, and the finally obtained rectangle is taken as the second rectangle.
  • the real-time image superimposing apparatus in the embodiment of the present invention and the real-time image superimposing method introduced in the foregoing section are based on two aspects under the same inventive concept, and the specific processing procedure of the real-time image superimposing method has been described in detail in the foregoing section. For the sake of simplicity of the description, it will not be described again.
  • the plane in the current key frame and the longest edge line on each plane are determined, and the virtual image is superimposed to the first in the current key frame.
  • the bonding surface of the main feature in the virtual image is adhered to the first plane, and the positive side of the main feature is set to be parallel or perpendicular to the edge of the first plane, so there is no need to
  • the digital environment of the real environment can provide users with the visual experience of AR technology, reducing the user's experience to the AR technology's pre-steps, real-time performance, and the scope of application is greatly improved, and also combines the actual edge of the plane in the real environment.
  • the authenticity of the main features in the virtual image is significantly improved, thereby reducing or avoiding the possibility that the user perceives that the main feature is virtual, and then can provide an immersive experience to the user.
  • the technical solution of adjusting the size of the main feature is such that the superimposed virtual
  • the authenticity of the main features in the image is further enhanced, thereby further reducing or avoiding the possibility that the user perceives that the primary feature is virtual, and in turn is better able to provide an immersive experience to the user.
  • the invention is not limited to the specific embodiments described above.
  • the invention extends to any new features disclosed in this specification Sign or any new combination, as well as any new method or process steps or any new combination disclosed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种用于增强现实的实时图像叠加方法及装置。所述方法包括：获取一单目摄像机实时拍摄预设场景时的视频信息，并从视频信息中确定出关键帧（S1）；根据当前关键帧和当前关键帧的在前关键帧实时建立点云数据，所述当前关键帧和所述在前关键帧之间的视差角大于预设视差角，所述点云数据中的每一个数据点包括有三维坐标（S2）；将当前关键帧对应的点云数据投影到当前关键帧上，并根据投影结果以及当前关键帧的超像素分割结果，确定当前关键帧中的平面（S3）；根据所述当前关键帧中每一个平面对应的点云数据，确定每一个平面上最长的边缘直线（S4）；在将虚拟图像叠加到当前关键帧中的第一平面上时，将虚拟图像中主要特征物的贴合面与第一平面贴合，且将主要特征物的正向边设置为与第一平面的边缘直线平行或者垂直，正向边为所述主要特征物的预设朝向面和预设底面之间的相交线（S5）。采用该方法，无需预先对真实环境进行数字化处理即能够向用户提供AR技术的视觉体验，实时性较强。

Description

一种用于增强现实的实时图像叠加方法及装置
本申请要求享有2016年8月30日提交的名称为“一种用于增强现实的实时图像叠加方法及装置”的中国专利申请CN201610770762.9的优先权，其全部内容通过引用并入本文中。
技术领域
本发明涉及增强现实领域，尤其涉及一种用于增强现实的实时图像叠加方法及装置。
背景技术
增强现实(英文：Augmented Reality；简称：AR)是近年来机器视觉领域备受关注的热点研究之一。该技术的目的是在AR设备的屏幕上所呈现的由摄像机采集的真实环境中，通过特殊的计算机图像和视频处理算法，把虚拟图像叠加到真实环境的图像上，使得用户能够获得全新的超越现实的感官体验。在把虚拟图像叠加到真实环境的图像上时，需要获得真实环境的数字化模型，才能够保证虚拟图像叠加时的准确性。
目前，对真实环境的数字化处理一般需要预先处理完成。例如，先通过激光式扫描仪或照相式扫描仪建立点云数据，再对点云数据进行平面检测等相应的处理，对真实环境的数字化处理完成之后，才能够提供给AR技术使用。
可以看出，由于现有的AR技术中对真实环境的数字化需要预先进行处理，实时性较低，所以增加了用户体验到AR技术的前置步骤，也增加了用户花费的时间和精力，从而导致用户对AR技术的用户体验较低。
因此，现有的AR技术中存在因需要对真实环境的数字化需要预先进行处理，导致增加了用户体验到AR技术的前置步骤，以及增加了用户花费的时间和精力的技术问题。
发明内容
本发明的目的是提供一种用于增强现实的实时图像叠加方法以及装置，以解决现有的AR技术中存在的因需要对真实环境的数字化需要预先进行处理，导致增加了用户体验到AR技术的前置步骤，以及增加了用户花费的时间和精力的技术问题。
为了实现上述发明目的，本发明实施例第一方面提供了一种用于增强现实的实时图像叠加方法，包括：
获取一单目摄像机实时拍摄预设场景时的视频信息，并从所述视频信息中确定出关键帧；
根据当前关键帧和所述当前关键帧的在前关键帧实时建立点云数据，所述当前关键帧和所述在前关键帧之间的视差角大于预设视差角，所述点云数据中的每一个数据点包括有三维坐标；
将所述当前关键帧对应的点云数据投影到所述当前关键帧上，并根据投影结果以及所述当前关键帧的超像素分割结果，确定所述当前关键帧中的平面；
根据所述当前关键帧中每一个平面对应的点云数据，确定所述每一个平面上最长的边缘直线；
在将虚拟图像叠加到所述当前关键帧中的第一平面上时，将所述虚拟图像中主要特征物的贴合面与所述第一平面贴合，且将所述主要特征物的正向边设置为与所述第一平面的边缘直线平行或者垂直，所述正向边为所述主要特征物的预设朝向面和预设底面之间的相交线。
可选地，所述确定所述每一个平面上最长的边缘直线，具体包括：
按以下确定当前关键帧中第二平面上最长的边缘直线的步骤，来确定所述每一个平面上最长的边缘直线：
将所述第二平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第一平面图像，所述第一预设平面与所述第二平面平行；
对所述第一平面图像进行形态学处理，并根据处理结果确定所述第一平面图像的边缘图像；
对所述边缘图像进行边沿直线检测，并按预设条件选择直线；
确定满足所述预设条件的直线为所述第二平面上最长的边缘直线，并将满足所述预设条件的直线按逆投影的方式投影到所述第二平面中。
可选地，所述预设条件具体为：直线的长度最长和/或直线的两边预设范围内数据点数目之和的差最大。
可选地，在确定所述每一个平面上最长的边缘直线时，所述方法还包括：为每一个平面确定一个矩形，所述矩形的一边与其所在平面的边缘直线平行；
所述在将虚拟图像叠加到所述当前关键帧中的第一平面上时，所述方法还包括：根据所述贴合面与所述第三平面的矩形的相对大小，调整所述主要特征物的大小。
可选地，所述为每一个平面确定一个矩形，具体包括：
按以下确定当前关键帧中第三平面中的矩形的步骤，来确定所述每一个平面中的矩形：
将所述第三平面对应的点云数据按投影到第二预设平面的方式转换为二维坐标，获得第二平面图像，所述第二预设平面与所述第三平面平行；
确定所述第二平面图像中的种子点，所述种子点为所述第二平面图像中预设大小区域内的数据点占比最高的中心点；
根据所述种子点确定所述第二平面图像中的矩形；
将所述第二平面图像中的矩形的端点坐标按逆投影的方式转换为三维坐标，为所述第三平面确定出矩形。
可选地，所述确定所述第二平面图像中的种子点，具体包括：
计算所述第二平面图像中以每一个数据点为中心的预设大小区域的数据点占比；
若其中第一数据点的数据点占比最高，则将所述第一数据点作为所述种子点。
可选地，所述根据所述种子点确定所述第二平面图像中的矩形，具体包括：
设定以所述种子点为中心、面积等于第一阈值的第一矩形；
检测所述第一矩形与所述第三平面之间的共同面积；
若所述共同面积与所述第一矩形的面积之间的第一比值大于第二阈值，则按所述种子点到所述第一矩形的四条边的四个方向对所述第一矩形进行扩大，获得扩大后的第二矩形；
将所有种子点对应的第二矩形中的面积最大者作为所述第二平面图像中的矩形。
可选地，所述按所述种子点到所述第一矩形的四条边的四个方向对所述第一矩形进行扩大，获得扩大后的第二矩形，具体包括：
按所述种子点到所述第一矩形的四条边的方向分别进行扩大；
按每个方向扩大后重新检测所述第一比值；
若重新检测后的第一比值大于所述第二阈值，则在相同的方向上继续扩大，否则在相同的方向上停止扩大，并在其他的方向上扩大；
直到在每个方向上都停止扩大，将最后获得的矩形作为所述第二矩形。
本发明实施例第二方面提供了一种用于增强现实的实时图像叠加装置，包括：
关键帧确定单元，用于获取一单目摄像机实时拍摄预设场景时的视频信息，并从所述视频信息中确定出关键帧；
点云数据建立单元，用于根据当前关键帧和所述当前关键帧的在前关键帧实时建立点云数据，所述当前关键帧和所述在前关键帧之间的视差角大于预设视差角，所述点云数据中的每一个数据点包括有三维坐标；
平面确定单元，用于将所述当前关键帧对应的点云数据投影到所述当前关键帧上，并根据投影结果以及所述当前关键帧的超像素分割结果，确定所述当前关键帧中的平面；
边缘直线确定单元，用于根据所述当前关键帧中每一个平面对应的点云数据，确定所述每一个平面上最长的边缘直线；
图像叠加单元，用于在将虚拟图像叠加到所述当前关键帧中的第一平面上时，将所述虚拟图像中主要特征物的贴合面与所述第一平面贴合，且将所述主要特征物的正向边设置为与所述第一平面的边缘直线平行或者垂直，所述正向边为所述主要特征物的预设朝向面和预设底面之间的相交线。
可选地，所述边缘直线确定单元具体用于按以下确定当前关键帧中第二平面上最长的边缘直线的步骤，来确定所述每一个平面上最长的边缘直线：
将所述第二平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第一平面图像，所述第一预设平面与所述第二平面平行；
对所述第一平面图像进行形态学处理，并根据处理结果确定所述第一平面图像的边缘图像；
对所述边缘图像进行边沿直线检测，并按预设条件选择直线；
确定满足所述预设条件的直线为所述第二平面上最长的边缘直线，并将满足所述预设条件的直线按逆投影的方式投影到所述第二平面中。
可选地，所述预设条件具体为：直线的长度最长和/或直线的两边预设范围内数据点数目之和的差最大。
可选地，所述实时图像叠加装置还包括矩形确定单元，所述矩形确定单元用于在所述边缘直线确定单元确定所述每一个平面上最长的边缘直线时，为每一个平面确定一个矩形，所述矩形的一边与其所在平面的边缘直线平行；
所述图像叠加单元还用于在在将虚拟图像叠加到所述当前关键帧中的第一平面上时，根据所述贴合面与所述第一平面的矩形的相对大小，调整所述主要特征物的大小。
可选地，所述矩形确定单元具体用于按以下确定当前关键帧中第三平面中的矩形的步骤，来确定所述每一个平面中的矩形：
将所述第三平面对应的点云数据按投影到第二预设平面的方式转换为二维坐标，获得第二平面图像，所述第二预设平面与所述第三平面平行；
确定所述第二平面图像中的种子点，所述种子点为所述第二平面图像中预设大小区域内的数据点占比最高的中心点；
根据所述种子点确定所述第二平面图像中的矩形；
将所述第二平面图像中的矩形的端点坐标按逆投影的方式转换为三维坐标，为所述第三平面确定出矩形。
可选地，所述矩形确定单元具体用于通过以下步骤来确定所述第二平面图像中的种子点：计算所述第二平面图像中以每一个数据点为中心的预设大小区域的数据点占比，若其中第一数据点的数据点占比最高，则将所述第一数据点作为所述种子点。
可选地，所述矩形确定单元具体用于通过以下步骤来根据所述种子点确定所述第二平面图像中的矩形：设定以所述种子点为中心、面积等于第一阈值的第一矩形，并检测所述第一矩形与所述第三平面之间的共同面积，若所述共同面积与所述第一矩形的面积之间的第一比值大于第二阈值，则按所述种子点到所述第一矩形的四条边的四个方向对所述第一矩形进行扩大，获得扩大后的第二矩形，并将所有种子对应的第二矩形中的面积最大者作为所述第二平面图像中的矩形。
可选地，所述矩形确定单元具体用于通过以下步骤来获得扩大后的第二矩形：按所述种子点到所述第一矩形的四条边的方向分别进行扩大，并按每个方向扩大后重新检测所述第一比值，若重新检测后的第一比值大于所述第二阈值，则在相同的方向上继续扩大，否则在相同的方向上停止扩大，并在其他的方向上扩大，直到在每个方向上都停止扩大，将最后获得的矩形作为所述第二矩形。
本发明实施例中的一个或者多个技术方案，至少具有如下技术效果或者优点：
1、由于采用了在单目摄像机实时拍摄预设场景的过程中，确定当前关键帧中的平面以及每一个平面上最长的边缘直线，并在将虚拟图像叠加到当前关键帧中的第一平面时，将虚拟图像中主要特征物的贴合面与第一平面贴合，并且将主要特征物的正向边设置为与 第一平面的边缘直线平行或者垂直的技术方案，所以无需预先对真实环境进行数字化处理即能够向用户提供AR技术的视觉体验，减少了用户体验到AR技术的前置步骤，实时性较强，适用范围也大大提高，同时也结合了真实环境中平面的实际边缘，明显提高了虚拟图像中的主要特征物的真实性，从而减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够向用户提供沉浸式体验。
2、由于采用了确定当前关键帧中的平面中的矩形，并根据虚拟图像中主要特征物的贴合面与矩形的相对大小，调整所述主要特征物的大小的技术方案，使得叠加的虚拟图像中主要特征物的真实性进一步提高，从而进一步减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够更好地向用户提供沉浸式体验。
附图说明
图1为本发明实施例提供的用于增强现实的实时图像叠加方法的流程示意图；
图2为本发明实施例提供的单目摄像机实时拍摄预设场景时的示意图；
图3为本发明实施例提供的用于增强现实的实时图像叠加装置的模块示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明实施例提供一种用于增强现实的实时图像叠加方法。请参考图1，图1为本发明实施例提供的用于增强现实的实时图像叠加方法的流程示意图。如图1所示，该方法主要包括步骤S1至步骤S5。
在步骤S1中，获取一单目摄像机实时拍摄预设场景时的视频信息，并从视频信息中确定出关键帧。
具体来讲，单目摄像机例如可以设置在头戴式显示设备或者眼镜式显示设备上，在此不做限制。通过该单目摄像机在移动并且旋转的过程中拍摄预设场景，即能够获得相应的视频信息。需要说明的是，该单目摄像机在移动并且旋转的过程中进行拍摄的原因是为了保证后续步骤对视差角的需求。在获取到视频信息后，即可以通过一定的策略来确定视频信息中的关键帧。例如可以在关键帧与关键帧之间的间隔过远时确定新的关键帧，或者也 可以通过像素帧平均法、直方图帧平均法或者两者的混合来确定视频信息中的关键帧。当然还可以使用其他方法，在此不做限制。一般来讲，选取的准则是优先考虑关键帧之间的不相似性，即以帧之间相似度作为衡量依据，每次寻找关键帧都保证各关键帧之间具有最小相似度，以使关键帧带有最大信息量，在此就不再赘述了。
在步骤S2中，根据当前关键帧和当前关键帧的在前关键帧实时建立点云数据，当前关键帧和在前关键帧之间的视差角大于预设视差角，点云数据中的每一个数据点包括有三维坐标。
具体来讲，由于当前关键帧和在前关键帧之间的视差角大于预设视差角，所以可以通过当前关键帧和在前关键帧之间进行立体匹配而求得视差角，再根据三角测量原理，即能够恢复当前关键帧对应的点云数据中每个点的三维信息，也即点云数据中的每一个数据点包括有三维坐标。
需要说明的是，预设视差角若较小，则容易受到噪声影响，从而影响点云数据中每一个数据点的三维坐标的精度；预设视差角若较大，则会导致匹配困难，从而无法建立点云数据。因此，预设视差角的具体数值可以由本领域所属的技术人员根据实际情况进行设置，以满足实际情况的需求，在此就不再赘述了。
在步骤S3中，将当前关键帧对应的点云数据投影到当前关键帧上，并根据投影结果以及当前关键帧的超像素分割结果，确定当前关键帧中的平面。
具体来讲，也即对当前关键帧进行超像素分割，获得超像素分割结果，并将步骤S2中获得的点云数据投影到当前关键帧上，获得投影结果。这样，当前关键帧上的每个超像素上均存在投影点，这样，即能够通过每个超像素关联的投影点，计算出每个超像素对应的平面，从而能够确定当前关键帧中的平面。在需要将虚拟图像叠加到当前关键帧上时，即能够根据计算获得的平面，选择具体叠加到当前关键帧上的某一个平面上，选择的方式在此就不做介绍了。
在步骤S4中，根据当前关键帧中每一个平面对应的点云数据，确定每一个平面上最长的边缘直线。
具体来讲，在通过步骤S3确定当前关键帧中的平面后，当前关键帧中的每一个平面均会对应于点云数据中的一部分，这样，即能够根据每一个平面对应的点云数据，通过边缘检测和直线检测等算法，确定每一个平面上最长的边缘直线。
在步骤S5中，在将虚拟图像叠加到当前关键帧中的第一平面上时，将虚拟图像中主要特征物的贴合面与第一平面贴合，且将主要特征物的正向边设置为与第一平面的边缘直 线平行或者垂直。这里，正向边为主要特征物的预设朝向面和预设底面之间的相交线。
在实际应用中，AR体验一般可以通过两种方式提供给用户。第一种是通过摄像机摄录外界环境，并在摄录的影像信息上叠加虚拟图像，然后将叠加后的图像传递到人眼。第二种是通过透镜将预设场景的光线传递至人眼，同时将虚拟图像的光线投射到透镜上，使得虚拟图像的光线融合到预设场景的光线中，再传递到人眼。
在步骤S5中，具体来讲，以虚拟图像中的主要特征物为一个虚拟的玩偶为例。若需要将该虚拟的玩偶叠加到桌面(该桌面与地面平行)，则可以将该玩偶的贴合面设置为玩偶的底面，同时还将该玩偶的正向边设置为与桌面的边缘直线平行或者垂直，该正向边即是该玩偶的正面与底面的相交线。这样，在将一个虚拟的玩偶叠加到桌子的侧面后，该玩偶的底面即会贴合在桌面上，并且该玩偶的正向边会平行或垂直于桌面的边缘直线。这样，该玩偶看起来就像一个真实的玩偶被树立在桌子的桌面上，并且该玩偶的正面朝向桌面的实际边缘或者垂直于桌面的实际边缘，用户通过增强现实设备看到该玩偶，就不会立即察觉到该玩偶为虚拟的，玩偶的真实性有了明显的提高，从而减少或者避免了用户察觉到该主要特征物为虚拟的可能性。
可以看出，由于采用了在单目摄像机实时拍摄预设场景的过程中，确定当前关键帧中的平面以及每一个平面上最长的边缘直线，并在将虚拟图像叠加到当前关键帧中的第一平面时，将虚拟图像中主要特征物的贴合面与第一平面贴合，并且将主要特征物的正向边设置为与第一平面的边缘直线平行或者垂直的技术方案，所以无需预先对真实环境进行数字化处理即能够向用户提供AR技术的视觉体验，减少了用户体验到AR技术的前置步骤，实时性较强，适用范围也大大提高，同时也结合了真实环境中平面的实际边缘，明显提高了虚拟图像中的主要特征物的真实性，从而减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够向用户提供沉浸式体验。
在接下来的部分中，将结合具体附图，来详细介绍上述技术方案。
在步骤S1中，获取一单目摄像机实时拍摄预设场景时的视频信息，并从视频信息中确定出关键帧。
具体来讲，也即通过单目摄像机在移动且旋转的过程中，实时拍摄该预设场景，并将拍摄获得的视频信息传递到存储器如内存中。请继续参考图2，图2为本发明实施例提供的单目摄像机实时拍摄预设场景时的示意图。如图2所示，以物体201为预设场景中的一部分为例，该物体201可以为桌子、墙面或地面等等。单目摄像机202在移动且旋转的过程中实时拍摄该预设场景中的物体201。
在接下来的部分中，请继续参考图2，将以单目摄像机202分别在两个不相同的位置A1和A2实时拍摄预设场景中的物体201的过程为例进行说明。单目摄像机202在A1点获得的图像帧为F1，在A2点获得的图像帧为F2，并设定F1和F2为从视频信息提取出的关键帧。其中，F2为当前关键帧，F1为F2的在前关键帧。在实际应用中，F1和F2可以相邻，也可以不相邻，在此不做限制。
需要说明的是，在实际应用中，单目摄像机实时拍摄预设场景和从视频信息中确定出关键帧这两个步骤是同时进行的，并不是视频拍摄完成之后才进行从视频信息中确定出关键帧这一步骤。
在通过步骤S1确定出视频信息的关键帧后，本发明实施例提供的实时图像叠加方法进入步骤S2：根据当前关键帧和当前关键帧的在前关键帧实时建立点云数据，当前关键帧和在前关键帧之间的视差角大于预设视差角，点云数据中的每一个数据点包括有三维坐标。
具体来讲，步骤S2可以包括：通过对当前关键帧和在前关键帧进行立体匹配，获得当前关键帧和在前关键帧之间的视差角；根据三角测量原理，实时建立点云数据。
请继续参考图2，如图2所示，以真实环境中物体201上的一点P12为例，单目摄像机202在A1点拍摄的关键帧F1中，实际点P12为点P1，在A2点拍摄的关键帧F2中，实际点P12对应为点P2。这样，通过对F1和F2进行立体匹配后，即能够获得关键帧F1和关键帧F2的视差角，也即能够获得点P1、点P2和点P12构成的三角形。根据三角测量原理，求解点P1、点P2和点P12构成的三角形，也就能够获得点P12的坐标。同理，只要真实环境中物体201上同时在关键帧F1和关键帧F2上成像并进行匹配的点(这一类点一般被称为特征点)均能够获得其坐标，这样，即能够建立包括真实环境中物体201上所有特征点在内的点云数据。
需要说明的是，此处获得的点云数据是根据关键帧F1和关键帧F2而获得的，是一个半稠密(英文：Semi-Dense)的点云数据。半稠密的点云数据中数据点的稠密程度介于稀疏的点云数据和稠密的点云数据之间，同时，该点云数据中每一个点的坐标均是相对坐标，但每个点之间的相对位置是固定的。
通过本实施例的介绍，本领域所属的技术人员还能够根据实际情况，采用其他的单目并发建图与定位算法(英文：simultaneous localization and mapping；简称：SLAM)来获得点云数据，例如ORB(英文：Oriented FAST and Rotated BRIEF；中文：面向快速和简单旋转)-SLAM算法或PTAM(英文：Parallel Tracking and Mapping；中文：并行跟踪与 绘图)算法等等，在此不做限制。
在通过步骤S2获得点云数据之后，本发明实施例提供的实时图像叠加方法进入步骤S3，即：将当前关键帧对应的点云数据投影到当前关键帧上，并根据投影结果以及当前关键帧的超像素分割结果，确定当前关键帧中的平面。
具体来讲，步骤S3可以包括：对当前关键帧进行超像素分割处理，获得当前关键帧的超像素集合；将当前关键帧对应的点云数据投影到当前关键帧上，获得与数据点一一对应的投影点；遍历所有投影点，检测每一个投影点到每一个超像素的最短距离，并且，在第一投影点到第一超像素的最短距离小于第一预设值时，将第一投影点对应的第一数据点与第一超像素建立关联；根据每个超像素关联的数据点，获得每个超像素对应的平面方程，从而获得当前关键帧中所有超像素对应的平面方程。
首先，介绍一下超像素分割的过程：超像素是指具有相似的灰度、纹理、颜色或亮度等特征的相邻像素构成的有一定视觉意义的不规则像素块，而超像素分割是将一幅是像素级(英文：pixel-level)的图，划分为区域级(英文：district-level)的图，每个子区域的某个特征具有很强的一致性，例如每个子区域的灰度很接近。在实际应用中，超像素的分割算法又被称为生成算法，超像素的分割算法有很多种，基本可以分为基于图论法和基于梯度下降法两大类。基于图论法的超像素分割算法例如可以包括基于图的图像分割算法(英文：graph-based Image Segmentation)、超像素格算法(英文：superpixel lattices)等等。基于梯度下降法的超像素分割算法例如可以包括分水岭算法(英文：watershed)、均值漂移算法(英文：mean-shift)或简单线性迭代聚类算法(英文：simple linear iterative cluster；简称：SLIC)等等，在此就不再赘述了。
在通过超像素分割算法对当前关键帧进行超像素分割处理后，即能够获得当前关键帧的超像素集合，超像素集合中的每个超像素均包括一个或者多个像素。由于每个超像素代表着一个低纹理、低梯度值、颜色相近的区域，所以可以近似地认为每个超像素都是一个平面区域。
接着，将当前关键帧对应的点云数据投影到当前关键帧上，这样，点云数据中的每个数据点都能够对应于一个投影点，每个投影点均为当前关键帧所在平面上的二维点，投影的具体过程在此就不再赘述了。
然后，遍历所有投影点，检测每一个投影点到每一个超像素的最短距离。例如，以当前关键帧所在平面的任意点为原点建立坐标系，即能够获得所有投影点的坐标值，以及所有超像素的质心坐标和边缘线坐标。这样，针对第一投影点而言，即能够计算出第一投影 点距离每一个超像素的最短距离。具体计算方法可以如下：通过连接线来分别连接第一投影点和所有超像素的质心，则第一投影点与该连接线和每个超像素的边缘的交点之间的距离为最短距离。
当然，在其他实施例中，本领域所属的技术人员能够根据实际情况，采用其他合适的计算方法，以满足实际情况的需要，在此就不再赘述了。
这样，若计算出第一投影点与某个超像素例如第一超像素的最短距离小于第一预设值，第一预设值例如可以为1个像素或者3个像素，则可以将点云数据中与第一投影点对应的数据点与第一超像素建立关联，以此类推，直到当前关键帧对应的点云数据中所有数据点均与当前关键帧中的超像素建立关联。
最后，根据每个超像素关联的数据点，即能够获得该超像素对应的平面方程。
在具体实施过程中，为了保证所获得的该超像素对应的平面方程的准确性，需要先对每个超像素关联的数据点进行处理。在本实施例中，可以通过随机抽样一致性算法(英文：RANdom Sample Consensus；简称：RANSAC)，来去除每个超像素关联的数据点中的外点，具体处理过程如下：
以第一超像素关联的数据点为例，可以从其中随机抽取一定数量的样本数据点，例如可以从其中抽取5个或者10个样本数据点。在本实施例中，设定抽取的样本数据点的数目为m个。这样，由于每个样本数据点包括三维坐标，则可以建立一个m*3的样本矩阵，在本实施例中，设定该样本矩阵为Am*3。然后对Am*3进行奇异值分解，即能够将矩阵Am*3分解为A＝UΣVT，其中U是一个m*m的方阵，Σ是m*3的矩阵，Σ矩阵中对角线上的元素称为奇异值，除了奇异值外其他元素都为0，VT是一个3*3的方阵，称为奇异向量，最小奇异值对应的奇异向量就是该第一超像素对应的平面方程的法向量，代入上述任意一个样本数据点即能够获得第一超像素对应的平面方程，在此就不再赘述了。第一超像素对应的平面方程具体可以为Ax+By+Cz+D＝0，设定此平面方程为第一平面方程。当然，在具体实施过程中，可以先计算出抽取的样本数据点的质心坐标，然后将其代入到第一超像素对应的平面的法向量中，从而获得更精确的平面方程，在此就不再赘述了。
接着，计算第一超像素关联的数据点到第一平面方程对应的平面的距离。例如，以第一超像素关联的数据点中的一点Pj(Xj，Yj，Zj)为例，Pj到第一平面方程对应的平面的距离为：
Figure PCTCN2017098476-appb-000001
在获得第一超像素关联的所有数据点到第一平面方程对应的平面的距离后，即可以将 与第一平面方程对应的平面之间的距离小于第一设定值的数据点设为内点，以及将与第一平面方程对应的平面之间的距离大于等于第一设定值的数据点设为外点，这样，即能够从第一超像素关联的数据点中剔除外点，再根据剔除了外点的第一超像素关联的数据点，重新计算第一超像素对应的平面方程，具体的计算过程如前，在此就不再赘述了。设定此次计算所获得的平面方程为第二平面方程。由于本次计算第二平面方程时剔除了外点，从而能够有效地提高计算所获得的平面方程的精度，在此就不再赘述了。
当然，此时还可以再次计算第一超像素关联的数据点到第二平面方程的距离，将与该平面方程的距离小于第一设定值的数据点设为内点，且统计该子数据集中属于第二拟合平面的内点数目，并计算该子数据集内所有数据点到该平面方程的平均距离为
Figure PCTCN2017098476-appb-000002
以供后续步骤使用。
这样，循环执行上述从第一超像素关联的数据点中抽取样本数据点计算第一平面方程、以及剔除外点后再次计算第二平面方程的步骤，就能够得到多个第二平面方程。由于每次随机抽取的样本数据点均不一致，所以每次循环最终得到的第二平面方程均会存在一定的差异。一般说来，每次循环获得的第二平面方程的内点数目都是不相同的，因此，此时可以将N个第二平面方程中内点数目最多的确定为第一超像素对应的平面方程，也即该平面方程能够在最大程度上适用于第一超像素关联的所有数据点。
需要说明的是，若存在有多个第二平面方程的内点数目相同，则可以选择所有数据点到平面方程对应的平面的平均距离最小的平面。若还是存在有多个第二平面方程满足条件，则可以进一步选择该子数据集内所有数据点到第二平面方程对应的平面的距离的方差最小的。当然，通过本实施例的介绍，本领域所属的技术人员还能够实际情况，选择其他合适的方法来选择第二平面方程，在此就不再赘述了。
需要说明的是，如果设定循环执行的次数越多，则最后获得的第二平面方程的准确性也就越高，但是如果循环执行的次数过多，就会导致计算量增大，从而会增加计算时间，因此本领域所属的技术人员在实际应用中，能够根据实际情况，设定循环执行的次数为一个合适数值，以满足实际情况的需要，在此就不再赘述了。
同理，可以计算获得当前关键帧中所有超像素各自对应的平面方程，从而能够确定当前关键帧中的平面。
在具体实施过程中，为了避免获得错误的平面方程，在获得每个超像素对应的平面方程之后，本发明实施例提供的实时图像叠加方法还包括：
针对每个超像素进行以下处理：将第一超像素的边缘投影到与当前关键帧相邻的N帧关键帧上，并计算N帧关键帧的每一关键帧中匹配的超像素的边缘的投影误差；在某一帧关键帧对应的投影误差小于第二预设值时，确定第一超像素对应的平面方程具有一个匹配；在第一超像素对应的平面方程具有超过第一预设数目的匹配时，确定第一超像素对应的平面方程为正确的方程。
具体来讲，以当前关键帧中的第一超像素为例，根据实时建立的点云数据以及第一超像素最终确定的平面方程，即能够获得第一超像素的边缘，从而能够将第一超像素的边缘投影到与当前关键帧相邻的N帧关键帧上。例如，在N为2时，则可以将第一超像素的边缘投影到与当前关键帧的前一关键帧和后一关键帧上，也可以将第一超像素的边缘投影到与当前关键帧的前两帧关键帧或后两帧关键帧上。在N为4时，则可以将第一超像素的边缘投影到当前关键帧的前两帧关键帧和后两帧关键帧上，也可以将第一超像素的边缘投影到当前关键帧的前三帧关键帧和后一关键帧上，还可以将第一超像素的边缘投影到与当前关键帧的前一关键帧和后三帧关键帧上，在此不做限制。本领域所属的技术人员能够根据实际情况，选择合适的关键帧进行投影，以满足实际情况的需要，在此就不再赘述了。
接着，计算每一关键帧中匹配的超像素的边缘的投影误差，可以通过如下的表达式进行计算：
Figure PCTCN2017098476-appb-000003
其中，F(Tj,pu)为第j关键帧中Pu对应的投影点，
Figure PCTCN2017098476-appb-000004
为第j关键帧中超像素si距离投影点F(Tj,pu)最近的像素。
然后，判断每一帧的超像素的边缘的投影误差是否小于第二预设值，第二预设值例如可以是1个像素或者3个像素等等。在具体实施过程中，本领域所属的技术人员能够根据实际情况，设置第二预设值的具体数值，以满足实际情况的需要，在此就不再赘述了。
最后，在某一关键帧对应的投影误差小于第二预设值时，则确定第一超像素对应的平面方程具有一个匹配，这样第一超像素对应的平面方程具有超过第一预设数目的匹配时，则可以确定第一超像素对应的平面方程为正确的方程，也就是说，当前关键帧中超像素的平面方程能够适用于与相邻的关键帧中匹配的超像素，即表明当前关键帧中超像素的平面方程为正确的方程。
此时，相当于完成了对当前关键帧中每一个平面的稠密点云重建。
在通过步骤S3确定当前关键帧中的平面后，本发明实施例提供的实时图像叠加方法进入步骤S4，即：根据当前关键帧中每一个平面对应的点云数据，确定每一个平面上最长的边缘直线。
具体来讲，以当前关键帧中的第二平面为例进行说明，步骤S4具体可以包括步骤S41 至步骤S44。
在步骤S41中，将第二平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第一平面图像，第一预设平面与第二平面平行。
例如，可以先确定第一预设平面，该第一预设平面与第二平面平行。在第二平面的平面方程已知的情况下，可以很容易确定第一预设平面，在此不再赘述。然后，将第二平面对应的点云数据中的所有数据点投影到第一预设平面上，这样，就将第二平面对应的点云数据中的数据点的三维坐标转换为了二维坐标，从而获得了第一平面图像。
在步骤S42中，对第一平面图像进行形态学处理，并根据处理结果确定第一平面图像的边缘图像。
在具体实施过程中，可以是对第一平面图像进行膨胀腐蚀等处理，去除该二维图像中的毛刺，再对处理后的第一平面图像进行边缘检测。例如可以通过具有多个方向的多方位形态学结构元素，应用基本形态运算，得到多个方向的边缘检测结果，再把这些结果进行归一化运算、加权求和，即能够得到最终的图像边缘。方向的数目选取可以由本领域所属的技术人员根据实际情况进行设置，以满足实际情况的需要，在此就不再赘述了。
在步骤S43中，对边缘图像进行边沿直线检测，并按预设条件选择直线。
在具体实施过程中，可以是对边缘图像通过霍夫变换(英文：Hough Transform)算法、基于Freeman(人名)准则的直线检测算法或尺蠖蠕行算法(英文：cankerworm crawlalgorithm)等等，从边缘图像中确定出直线。由于边缘图像中的边沿直线有多条，所以需要按照预设条件来选择直线。
在本实施例中，该预设条件可以是：直线的长度最长和/或直线的两边预设范围内数据点数目之和的差最大。具体来讲，“直线的长度最长”是指该直线为边缘图像中的最长边，也即该直线为边缘图像对应的平面的最显著特征。“直线的两边预设范围内数据点数目之和的差最大”是指该直线的两边预设范围内，一边数据点相对较多，而另一边数据点相对较少，满足此条件的直线也是边缘图像对应的平面的最显著特征。
所以，同时满足预设条件中两个要求或者满足其中任一条件，都能够确认该直线代表了真实环境中平面的实际边缘。
当然，在实际应用中，通过本实施例的介绍，本领域所属的技术人员可以根据实际情况，通过其他方式来确定该边缘图像的最长边缘直线，以满足实际情况的需要。
在步骤S44中，确定满足预设条件的直线为第二平面上最长的边缘直线，并将满足 预设条件的直线按逆投影的方式投影到第二平面中。
在具体实施过程中，在通过步骤S43确定边缘直线并根据预设条件对边缘直线进行判断后，即能够确定满足预设条件的直线为边缘直线，此时即能够通过逆投影的方式，将该边缘直线逆投影到第二平面对应的点云数据中，从而能够确定第二平面的实际边缘直线。
针对当前关键帧中的每一个平面，重复执行上述的步骤S41-S44，即能够确定每一个平面上最长的边缘直线，在此就不再详细介绍了。
在通过步骤S4确定每一个平面上最长的边缘直线后，本发明实施例提供的实时图像叠加方法进入步骤S5，即：在将虚拟图像叠加到当前关键帧中的第一平面上时，将虚拟图像中主要特征物的贴合面与第一平面贴合，且将主要特征物的正向边设置为与第一平面的边缘直线平行或者垂直，正向边为主要特征物的预设朝向面和预设底面之间的相交线。
在具体实施过程中，以虚拟图像中的主要特征物为一个虚拟的玩偶为例，若需要将该虚拟的玩偶叠加到桌面(该桌面与地面平行)，则可以将该玩偶的贴合面设置为玩偶的底面，同时还将该玩偶的正向边设置为与桌面的边缘直线平行或者垂直，该正向边即是该玩偶的正面与底面的相交线，这样，在将一个虚拟的玩偶叠加到桌子的侧面后，该玩偶的底面即会贴合在桌面上，并且该玩偶的正向边会平行或垂直于桌面的边缘直线，这样，该玩偶看起来就像一个真实的玩偶被树立在桌子的桌面上，并且该玩偶的正面朝向桌面的实际边缘或者垂直于桌面的实际边缘，用户通过增强现实设备看到该玩偶，就不会立即察觉到该玩偶为虚拟的，玩偶的真实性有了明显的提高，从而减少或者避免了用户察觉到该主要特征物为虚拟的可能性。
可以看出，由于采用了在单目摄像机实时拍摄预设场景的过程中，确定当前关键帧中的平面以及每一个平面上最长的边缘直线，并在将虚拟图像叠加到当前关键帧中的第一平面时，将虚拟图像中主要特征物的贴合面与第一平面贴合，并且将主要特征物的正向边设置为与第一平面的边缘直线平行或者垂直的技术方案，所以无需预先对真实环境进行数字化处理即能够向用户提供AR技术的视觉体验，减少了用户体验到AR技术的前置步骤，实时性较强，适用范围也大大提高，同时也结合了真实环境中平面的实际边缘，明显提高了虚拟图像中的主要特征物的真实性，从而减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够向用户提供沉浸式体验。
在具体实施过程中，在将虚拟图像叠加到当前关键帧的第一平面上时，若虚拟图像中主要特征物的大小远远超过了第一平面，例如，在一个小桌子上面叠加一个特别大虚拟的 玩偶，也即桌子的桌面远远小于该玩偶的底面，则也容易导致用户察觉到该主要特征物为虚拟的。因此，为了避免这一缺陷的出现，在通过步骤S4确定每一个平面上最长的边缘直线时，本发明实施例提供的实时图像叠加方法还包括：为每一个平面确定一个矩形，矩形的一边与其所在平面的边缘直线平行。同时，在将虚拟图像中主要特征物的贴合面与第一平面贴合时，本发明实施例提供的实时图像叠加方法还包括：根据贴合面与矩形的相对大小，调整主要特征物的大小。这样，即能够保证虚拟图像中的主要特征物的贴合面与待叠加的平面中的矩形之间的相对大小，调整虚拟图像中的主要特征物的大小，使得主要特征物的大小能够适应于待叠加的平面的大小，避免给用户以不真实的感觉。
在具体实施过程中，为每一个平面确定一个矩形，具体可以包括：按以下确定当前关键帧中第三平面中的矩形的步骤S61-S64，来确定每一个平面中的矩形。
在步骤S61中，将第三平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第二平面图像，第一预设平面与第三平面平行。
此处的投影过程与前述部分介绍的将第二平面对应的点云数据投影到第一预设平面的过程一致，在此就不再赘述了。
在步骤S62中，确定第二平面图像中的种子点，种子点为第二平面图像中预设大小区域内的数据点占比最高的中心点。
具体来讲，可以遍历第二平面图像中的每一个数据点，计算以每个数据点为中心的预设大小区域的数据点占比。例如，以预设大小区域为10*10的区域为例，以第一个数据点为中心的预设大小区域内的数据点有100个，以第二个数据点为中心的预设大小区域内的数据点有85个，以第三个数据点为中心的预设大小区域内的数据点有25个，则可以将第一个数据点作为种子点。
在具体实施过程中，预设大小区域的具体数值可以由本领域所属的技术人员更加实际情况进行设置，以满足实际情况的需要，在此不做限制。
在有多个数据点为中心的预设大小区域内的数据点占比相同的情况下，则可以继续扩大预设区域，直到数据点的数目变为预设数目，例如，5个或10个等等。
可以看出通过步骤S62选取的种子点不会在平面的边缘部分，从而保证了后续部分获得的矩形的正确性。
在步骤S63中，根据种子点确定第二平面图像中的矩形。
具体来讲，可以是先以种子点为中心设定第一矩形，该第一矩形的面积等于第一阈值。 第一阈值可以由本领域所属的技术人员根据实际情况进行设定，如平面的实际区域较大，则第一阈值可以较大，若平面较小，则第一阈值可以较小。例如第一阈值可以为10*12。接着，检测该第一矩形与第三平面之间的共同面积。然后，判断共同面积与第一矩形的面积之间的第一比值是否大于第二阈值。第二阈值也是本领域所属的技术人员能够根据实际情况设置的，例如第二阈值可以为80％、90％等等。在本实施例中，若第一比值大于第二阈值，则可以按种子点到第一矩形的四条边的四个方向对第一矩形进行扩大。
需要说明的是，将第二阈值设置为小于100％，是为了求出平面中的最大矩形，保证能够虚拟图像中的主要特征物能够以最大的面积或者体积展示给用户，从而能够向用户展示更多的细节。
在本实施例中，可以按照如下方式对第一矩形进行扩大：按种子点到第一矩形的四条边的方向分别进行扩大；按每个方向扩大后重新检测第一比值；若重新检测后的第一比值大于第二阈值，则在对应的方向上继续扩大，否则(即第一比值小于或者等于第二阈值)在对应的方向上停止扩大，并在其他的方向上扩大；直到在每个方向上都停止扩大，将最后获得的矩形作为第二矩形。
对每一个种子点都按上述过程进行处理后，就能够获得每个种子点对应的第二矩形，将其中面积最大的第二矩形作为第二平面图像中的矩形。
在步骤S64中：将第二平面图像中的矩形的端点坐标按逆投影的方式转换为三维坐标，为第三平面确定出矩形。
具体来讲，将矩形的四个端点坐标，分别按照步骤S61中投影的逆过程进行处理，从而获得具有三维坐标的四个端点，从而为第三平面，也即桌子的侧面确定出了矩形。
这样，在将虚拟图像中主要特征物的贴合面与第一平面贴合时，本发明实施例提供的图像叠加方法还包括：根据贴合面与矩形的相对大小，按比例调整主要特征物的大小。
具体来讲，虚拟图像中主要特征物的贴合面也是具有一定大小的，若贴合面小于矩形，则可以按照虚拟图像的原始大小，将主要特征物贴合在该平面的矩形上，也可以将虚拟图像的原始大小按预设比例变大，使得主要特征物在变大的同时，主要特征物的贴合面不大于该平面的矩形；若贴合面大于矩形，则可以按照虚拟图像按比例变小，使得主要特征物的贴合面小于或者等于矩形，当然，变小的比例需要设定一个最小值，否则会影响主要特征物的显示，若主要特征物的贴合面按最小值进行变小调整后，仍然大于或者远远大于该平面的矩形面，则可以选择其他平面进行图像叠加，在此就不再赘述了。
这样，在将虚拟图像中的主要特征物的贴合面贴合在平面的矩形上时，不会由于贴合 面大于或者远远大于平面而使得主要特征物看起来是悬浮在空中，使得叠加的虚拟图像中主要特征物的真实性进一步提高，从而进一步减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够更好地向用户提供沉浸式体验。
可以看出，由于采用了确定当前关键帧中的平面中的矩形，并根据虚拟图像中主要特征物的贴合面与矩形的相对大小，调整所述主要特征物的大小的技术方案，使得叠加的虚拟图像中主要特征物的真实性进一步提高，从而进一步减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够更好地向用户提供沉浸式体验。
基于同一发明构思，本发明实施例还提供一种用于增强现实的实时图像叠加装置。请参考图3，图3为本发明实施例提供的用于增强现实的实时图像叠加装置的模块示意图。如图3所示，该实时图像叠加装置主要包括关键帧确定单元301、点云数据建立单元302、平面确定单元303、边缘直线确定单元304和图像叠加单元305。
具体地，关键帧确定单元301，用于获取一单目摄像机实时拍摄预设场景时的视频信息，并从所述视频信息中确定出关键帧。
点云数据建立单元302，用于根据当前关键帧和所述当前关键帧的在前关键帧实时建立点云数据，所述当前关键帧和所述在前关键帧之间的视差角大于预设视差角，所述点云数据中的每一个数据点包括有三维坐标。
平面确定单元303，用于将所述当前关键帧对应的点云数据投影到所述当前关键帧上，并根据投影结果以及所述当前关键帧的超像素分割结果，确定所述当前关键帧中的平面。
边缘直线确定单元304，用于根据所述当前关键帧中每一个平面对应的点云数据，确定所述每一个平面上最长的边缘直线。
图像叠加单元305，用于在将虚拟图像叠加到所述当前关键帧中的第一平面上时，将所述虚拟图像中主要特征物的贴合面与所述第一平面贴合，且将所述主要特征物的正向边设置为与所述第一平面的边缘直线平行或者垂直，所述正向边为所述主要特征物的预设朝向面和预设底面之间的相交线。
在具体实施过程中，所述边缘直线确定单元304具体用于按以下确定当前关键帧中第二平面上最长的边缘直线的步骤，来确定每一个平面上最长的边缘直线：
将第二平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第一平面图像，所述第一预设平面与所述第二平面平行；
对所述第一平面图像进行形态学处理，并根据处理结果确定所述第一平面图像的边缘图像；
对所述边缘图像进行边沿直线检测，并按预设条件选择直线；
确定满足所述预设条件的直线为所述第二平面上最长的边缘直线，并将满足所述预设条件的直线按逆投影的方式投影到所述第二平面中。
在具体实施过程中，所述预设条件具体为：直线的长度最长和/或直线的两边预设范围内数据点数目之和的差最大。
在具体实施过程中，所述实时图像叠加装置还包括矩形确定单元306，所述矩形确定单元306用于在所述边缘直线确定单元304确定所述每一个平面上最长的边缘直线时，为每一个平面确定一个矩形，所述矩形的一边与其所在平面的边缘直线平行。
所述图像叠加单元305还用于在所述将所述虚拟图像中主要特征物的贴合面与所述第三平面贴合时，根据所述贴合面与所述第三平面的矩形的相对大小，调整所述主要特征物的大小。
在具体实施过程中，所述矩形确定单元306具体用于按以下确定当前关键帧中第三平面中的矩形的步骤，来确定每一个平面中的矩形：
将第三平面对应的点云数据按投影到第二预设平面的方式转换为二维坐标，获得第二平面图像，所述第二预设平面与所述第三平面平行；
确定所述第二平面图像中的种子点，所述种子点为所述第二平面图像中预设大小区域内的数据点占比最高的中心点；
根据所述种子点确定所述第二平面图像中的矩形；
将所述第二平面图像中的矩形的端点坐标按逆投影的方式转换为三维坐标，为所述第三平面确定出矩形。
在具体实施过程中，所述矩形确定单元306具体用于通过以下步骤来确定第二平面图像中的种子点：计算所述第二平面图像中以每一个数据点为中心的预设大小区域的数据点占比，若其中第一数据点的数据点占比最高，则将所述第一数据点作为所述种子点。
在具体实施过程中，所述矩形确定单元306具体用于通过以下步骤来根据所述种子点确定第二平面图像中的矩形：设定以所述种子点为中心、面积等于第一阈值的第一矩形，并检测所述第一矩形与所述第三平面之间的共同面积，若所述共同面积与所述第一矩形的面积之间的第一比值大于第二阈值，则按所述种子点到所述第一矩形的四条边的四个方向 对所述第一矩形进行扩大，获得扩大后的第二矩形，并将所有种子点对应的第二矩形中的面积最大者作为所述第二平面图像中的矩形。
在具体实施过程中，所述矩形确定单元306具体用于通过以下步骤来获得扩大后的第二矩形：按所述种子点到所述第一矩形的四条边的方向分别进行扩大，并按每个方向扩大后重新检测所述第一比值，若重新检测后的第一比值大于所述第二阈值，则在相同的方向上继续扩大，否则(即若所述第一比值小于或者等于所述第二阈值)在相同的方向上停止扩大，并在其他的方向上扩大，直到在每个方向上都停止扩大，将最后获得的矩形作为所述第二矩形。
本发明实施例中的实时图像叠加装置与前述部分介绍的实时图像叠加方法是基于同一发明构思下的两个方面，在前述部分中已经详细地介绍了实时图像叠加方法的具体处理过程，在此为了说明书的简洁，就不再赘述了。
本发明实施例中的一个或者多个技术方案，至少具有如下技术效果或者优点：
1、由于采用了在单目摄像机实时拍摄预设场景的过程中，确定当前关键帧中的平面以及每一个平面上最长的边缘直线，并在将虚拟图像叠加到当前关键帧中的第一平面时，将虚拟图像中主要特征物的贴合面与第一平面贴合，并且将主要特征物的正向边设置为与第一平面的边缘直线平行或者垂直的技术方案，所以无需预先对真实环境进行数字化处理即能够向用户提供AR技术的视觉体验，减少了用户体验到AR技术的前置步骤，实时性较强，适用范围也大大提高，同时也结合了真实环境中平面的实际边缘，明显提高了虚拟图像中的主要特征物的真实性，从而减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够向用户提供沉浸式体验。
2、由于采用了确定当前关键帧中的平面中的矩形，并根据虚拟图像中主要特征物的贴合面与矩形的相对大小，调整所述主要特征物的大小的技术方案，使得叠加的虚拟图像中主要特征物的真实性进一步提高，从而进一步减少或者避免了用户察觉到该主要特征物为虚拟的可能性，继而能够更好地向用户提供沉浸式体验。
本说明书中公开的所有特征，或公开的所有方法或过程中的步骤，除了互相排斥的特征和/或步骤以外，均可以以任何方式组合。
本说明书(包括任何附加权利要求、摘要和附图)中公开的任一特征，除非特别叙述，均可被其他等效或具有类似目的的替代特征加以替换。即，除非特别叙述，每个特征只是一系列等效或类似特征中的一个例子而已。
本发明并不局限于前述的具体实施方式。本发明扩展到任何在本说明书中披露的新特 征或任何新的组合，以及披露的任一新的方法或过程的步骤或任何新的组合。

Claims (16)

  1. 一种用于增强现实的实时图像叠加方法，其特征在于，包括：
    获取一单目摄像机实时拍摄预设场景时的视频信息，并从所述视频信息中确定出关键帧；
    根据当前关键帧和所述当前关键帧的在前关键帧实时建立点云数据，所述当前关键帧和所述在前关键帧之间的视差角大于预设视差角，所述点云数据中的每一个数据点包括有三维坐标；
    将所述当前关键帧对应的点云数据投影到所述当前关键帧上，并根据投影结果以及所述当前关键帧的超像素分割结果，确定所述当前关键帧中的平面；
    根据所述当前关键帧中每一个平面对应的点云数据，确定所述每一个平面上最长的边缘直线；
    在将虚拟图像叠加到所述当前关键帧中的第一平面上时，将所述虚拟图像中主要特征物的贴合面与所述第一平面贴合，且将所述主要特征物的正向边设置为与所述第一平面的边缘直线平行或者垂直，所述正向边为所述主要特征物的预设朝向面和预设底面之间的相交线。
  2. 如权利要求1所述的实时图像叠加方法，其特征在于，所述确定所述每一个平面上最长的边缘直线，具体包括：
    按以下确定当前关键帧中第二平面上最长的边缘直线的步骤，来确定所述每一个平面上最长的边缘直线：
    将所述第二平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第一平面图像，所述第一预设平面与所述第二平面平行；
    对所述第一平面图像进行形态学处理，并根据处理结果确定所述第一平面图像的边缘图像；
    对所述边缘图像进行边沿直线检测，并按预设条件选择直线；
    确定满足所述预设条件的直线为所述第二平面上最长的边缘直线，并将满足所述预设条件的直线按逆投影的方式投影到所述第二平面中。
  3. 如权利要求2所述的实时图像叠加方法，其特征在于，所述预设条件具体为：直线的长度最长和/或直线的两边预设范围内数据点数目之和的差最大。
  4. 如权利要求1-3中任一权项所述的实时图像叠加方法，其特征在于，
    在确定所述每一个平面上最长的边缘直线时，所述方法还包括：为每一个平面确定一个矩形，所述矩形的一边与其所在平面的边缘直线平行；
    所述在将虚拟图像叠加到所述当前关键帧中的第一平面上时，所述方法还包括：根据所述贴合面与所述第一平面的矩形的相对大小，调整所述主要特征物的大小。
  5. 如权利要求4所述的实时图像叠加方法，其特征在于，所述为每一个平面确定一个矩形，具体包括：
    按以下确定当前关键帧中第三平面中的矩形的步骤，来确定所述每一个平面中的矩形：
    将所述第三平面对应的点云数据按投影到第二预设平面的方式转换为二维坐标，获得第二平面图像，所述第二预设平面与所述第三平面平行；
    确定所述第二平面图像中的种子点，所述种子点为所述第二平面图像中预设大小区域内的数据点占比最高的中心点；
    根据所述种子点确定所述第二平面图像中的矩形；
    将所述第二平面图像中的矩形的端点坐标按逆投影的方式转换为三维坐标，为所述第三平面确定出矩形。
  6. 如权利要求5所述的实时图像叠加方法，其特征在于，所述确定所述第二平面图像中的种子点，具体包括：
    计算所述第二平面图像中以每一个数据点为中心的预设大小区域的数据点占比；
    若其中第一数据点的数据点占比最高，则将所述第一数据点作为所述种子点。
  7. 如权利要求5所述的实时图像叠加方法，其特征在于，所述根据所述种子点确定所述第二平面图像中的矩形，具体包括：
    设定以所述种子点为中心、面积等于第一阈值的第一矩形；
    检测所述第一矩形与所述第三平面之间的共同面积；
    若所述共同面积与所述第一矩形的面积之间的第一比值大于第二阈值，则按所述种子点到所述第一矩形的四条边的四个方向对所述第一矩形进行扩大，获得扩大后的第二矩形；
    将所有种子点对应的第二矩形中的面积最大者作为所述第二平面图像中的矩形。
  8. 如权利要求7所述的实时图像叠加方法，其特征在于，所述按所述种子点到所述 第一矩形的四条边的四个方向对所述第一矩形进行扩大，获得扩大后的第二矩形，具体包括：
    按所述种子点到所述第一矩形的四条边的方向分别进行扩大；
    按每个方向扩大后重新检测所述第一比值；
    若重新检测后的第一比值大于所述第二阈值，则在相同的方向上继续扩大，否则在相同的方向上停止扩大，并在其他的方向上扩大；
    直到在每个方向上都停止扩大，将最后获得的矩形作为所述第二矩形。
  9. 一种用于增强现实的实时图像叠加装置，其特征在于，包括：
    关键帧确定单元，用于获取一单目摄像机实时拍摄预设场景时的视频信息，并从所述视频信息中确定出关键帧；
    点云数据建立单元，用于根据当前关键帧和所述当前关键帧的在前关键帧实时建立点云数据，所述当前关键帧和所述在前关键帧之间的视差角大于预设视差角，所述点云数据中的每一个数据点包括有三维坐标；
    平面确定单元，用于将所述当前关键帧对应的点云数据投影到所述当前关键帧上，并根据投影结果以及所述当前关键帧的超像素分割结果，确定所述当前关键帧中的平面；
    边缘直线确定单元，用于根据所述当前关键帧中每一个平面对应的点云数据，确定所述每一个平面上最长的边缘直线；
    图像叠加单元，用于在将虚拟图像叠加到所述当前关键帧中的第一平面上时，将所述虚拟图像中主要特征物的贴合面与所述第一平面贴合，且将所述主要特征物的正向边设置为与所述第一平面的边缘直线平行或者垂直，所述正向边为所述主要特征物的预设朝向面和预设底面之间的相交线。
  10. 如权利要求9所述的实时图像叠加装置，其特征在于，所述边缘直线确定单元具体用于按以下确定当前关键帧中第二平面上最长的边缘直线的步骤，来确定所述每一个平面上最长的边缘直线：
    将所述第二平面对应的点云数据按投影到第一预设平面的方式转换为二维坐标，获得第一平面图像，所述第一预设平面与所述第二平面平行；
    对所述第一平面图像进行形态学处理，并根据处理结果确定所述第一平面图像的边缘图像；
    对所述边缘图像进行边沿直线检测，并按预设条件选择直线；
    确定满足所述预设条件的直线为所述第二平面上最长的边缘直线，并将满足所述预设条件的直线按逆投影的方式投影到所述第二平面中。
  11. 如权利要求10所述的实时图像叠加装置，其特征在于，所述预设条件具体为：直线的长度最长和/或直线的两边预设范围内数据点数目之和的差最大。
  12. 如权利要求9-11中任一权项所述的实时图像叠加装置，其特征在于，所述实时图像叠加装置还包括矩形确定单元，所述矩形确定单元用于在所述边缘直线确定单元确定所述每一个平面上最长的边缘直线时，为每一个平面确定一个矩形，所述矩形的一边与其所在平面的边缘直线平行；
    所述图像叠加单元还用于在将虚拟图像叠加到所述当前关键帧中的第一平面上时，根据所述贴合面与所述第一平面的矩形的相对大小，调整所述主要特征物的大小。
  13. 如权利要求12所述的实时图像叠加装置，其特征在于，所述矩形确定单元具体用于按以下确定当前关键帧中第三平面中的矩形的步骤，来确定所述每一个平面中的矩形：
    将所述第三平面对应的点云数据按投影到第二预设平面的方式转换为二维坐标，获得第二平面图像，所述第二预设平面与所述第三平面平行；
    确定所述第二平面图像中的种子点，所述种子点为所述第二平面图像中预设大小区域内的数据点占比最高的中心点；
    根据所述种子点确定所述第二平面图像中的矩形；
    将所述第二平面图像中的矩形的端点坐标按逆投影的方式转换为三维坐标，为所述第三平面确定出矩形。
  14. 如权利要求13所述的实时图像叠加装置，其特征在于，所述矩形确定单元具体用于通过以下步骤来确定所述第二平面图像中的种子点：
    计算所述第二平面图像中以每一个数据点为中心的预设大小区域的数据点占比；
    若其中第一数据点的数据点占比最高，则将所述第一数据点作为所述种子点。
  15. 如权利要求13所述的实时图像叠加装置，其特征在于，所述矩形确定单元具体用于通过以下步骤来根据所述种子点确定所述第二平面图像中的矩形：
    设定以所述种子点为中心、面积等于第一阈值的第一矩形；
    检测所述第一矩形与所述第三平面之间的共同面积；
    若所述共同面积与所述第一矩形的面积之间的第一比值大于第二阈值，则按所述种子点到所述第一矩形的四条边的四个方向对所述第一矩形进行扩大，获得扩大后的第二矩形；
    将所有种子点对应的第二矩形中的面积最大者作为所述第二平面图像中的矩形。
  16. 如权利要求15所述的实时图像叠加装置，其特征在于，所述矩形确定单元具体用于通过以下步骤来获得扩大后的第二矩形：
    按所述种子点到所述第一矩形的四条边的方向分别进行扩大；
    按每个方向扩大后重新检测所述第一比值；
    若重新检测后的第一比值大于所述第二阈值，则在相同的方向上继续扩大，否则在相同的方向上停止扩大，并在其他的方向上扩大；
    直到在每个方向上都停止扩大，将最后获得的矩形作为所述第二矩形。
PCT/CN2017/098476 2016-08-30 2017-08-22 一种用于增强现实的实时图像叠加方法及装置 WO2018040982A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610770762.9 2016-08-30
CN201610770762.9A CN107798704B (zh) 2016-08-30 2016-08-30 一种用于增强现实的实时图像叠加方法及装置

Publications (1)

Publication Number Publication Date
WO2018040982A1 true WO2018040982A1 (zh) 2018-03-08

Family

ID=61301502

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/098476 WO2018040982A1 (zh) 2016-08-30 2017-08-22 一种用于增强现实的实时图像叠加方法及装置

Country Status (2)

Country Link
CN (1) CN107798704B (zh)
WO (1) WO2018040982A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109445112A (zh) * 2019-01-05 2019-03-08 西安维度视界科技有限公司 一种ar眼镜及基于ar眼镜的增强现实方法
CN116912277A (zh) * 2023-09-12 2023-10-20 山东鲁泰化学有限公司 一种循环水除垢效果评估方法及系统
CN117893693A (zh) * 2024-03-15 2024-04-16 南昌航空大学 一种密集slam三维场景重建方法及装置

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI678679B (zh) * 2018-07-09 2019-12-01 財團法人資訊工業策進會 空間座標轉換伺服器以及方法
CN110969592B (zh) * 2018-09-29 2024-03-29 北京嘀嘀无限科技发展有限公司 图像融合方法、自动驾驶控制方法、装置和设备
CN109712249B (zh) * 2018-12-31 2023-05-26 成都纵横大鹏无人机科技有限公司 地理要素增强现实方法及装置
US10867447B2 (en) 2019-01-21 2020-12-15 Capital One Services, Llc Overlaying 3D augmented reality content on real-world objects using image segmentation
WO2020163038A1 (en) * 2019-02-07 2020-08-13 Interdigital Vc Holdings, Inc. Processing a point cloud
CN111695506B (zh) * 2020-06-11 2023-04-25 国网山东省电力公司电力科学研究院 一种输电线路风致异物短路故障预警方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130257904A1 (en) * 2012-04-03 2013-10-03 The Boeing Company System and Method for Virtual Engineering
US20150279083A1 (en) * 2014-03-26 2015-10-01 Microsoft Corporation Real-time three-dimensional reconstruction of a scene from a single camera
CN105023266A (zh) * 2014-04-29 2015-11-04 高德软件有限公司 增强现实实现方法、装置和终端设备
CN105678842A (zh) * 2016-01-11 2016-06-15 湖南拓视觉信息技术有限公司 室内环境三维地图制作方法及装置

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101189643A (zh) * 2005-04-25 2008-05-28 株式会社亚派 3d图像生成和显示系统
US20110102460A1 (en) * 2009-11-04 2011-05-05 Parker Jordan Platform for widespread augmented reality and 3d mapping
CN103582893B (zh) * 2011-06-08 2017-10-13 英派尔科技开发有限公司 用于增强现实表示的二维图像获取
WO2013029675A1 (en) * 2011-08-31 2013-03-07 Metaio Gmbh Method for estimating a camera motion and for determining a three-dimensional model of a real environment
AU2011253973B2 (en) * 2011-12-12 2015-03-12 Canon Kabushiki Kaisha Keyframe selection for parallel tracking and mapping
CN102831401B (zh) * 2012-08-03 2016-01-13 樊晓东 对无特定标记目标物体跟踪、三维叠加及交互的方法及系统
GB201303076D0 (en) * 2013-02-21 2013-04-10 Isis Innovation Generation of 3D models of an environment
US9674507B2 (en) * 2013-04-30 2017-06-06 Qualcomm Incorporated Monocular visual SLAM with general and panorama camera movements
CN103400409B (zh) * 2013-08-27 2016-08-10 华中师范大学 一种基于摄像头姿态快速估计的覆盖范围3d可视化方法
CN103500013B (zh) * 2013-10-18 2016-05-11 武汉大学 基于Kinect和流媒体技术的实时三维测图方法
CN105184825A (zh) * 2015-10-29 2015-12-23 丽水学院 一种面向室内场景的移动增强现实方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130257904A1 (en) * 2012-04-03 2013-10-03 The Boeing Company System and Method for Virtual Engineering
US20150279083A1 (en) * 2014-03-26 2015-10-01 Microsoft Corporation Real-time three-dimensional reconstruction of a scene from a single camera
CN105023266A (zh) * 2014-04-29 2015-11-04 高德软件有限公司 增强现实实现方法、装置和终端设备
CN105678842A (zh) * 2016-01-11 2016-06-15 湖南拓视觉信息技术有限公司 室内环境三维地图制作方法及装置

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109445112A (zh) * 2019-01-05 2019-03-08 西安维度视界科技有限公司 一种ar眼镜及基于ar眼镜的增强现实方法
CN116912277A (zh) * 2023-09-12 2023-10-20 山东鲁泰化学有限公司 一种循环水除垢效果评估方法及系统
CN116912277B (zh) * 2023-09-12 2023-12-12 山东鲁泰化学有限公司 一种循环水除垢效果评估方法及系统
CN117893693A (zh) * 2024-03-15 2024-04-16 南昌航空大学 一种密集slam三维场景重建方法及装置
CN117893693B (zh) * 2024-03-15 2024-05-28 南昌航空大学 一种密集slam三维场景重建方法及装置

Also Published As

Publication number Publication date
CN107798704A (zh) 2018-03-13
CN107798704B (zh) 2021-04-30

Similar Documents

Publication Publication Date Title
WO2018040982A1 (zh) 一种用于增强现实的实时图像叠加方法及装置
US11995902B2 (en) Facial signature methods, systems and software
US11107232B2 (en) Method and apparatus for determining object posture in image, device, and storage medium
CN109561296B (zh) 图像处理装置、图像处理方法、图像处理系统和存储介质
TWI709107B (zh) 影像特徵提取方法及包含其顯著物體預測方法
CN107798702B (zh) 一种用于增强现实的实时图像叠加方法以及装置
US20180218507A1 (en) 3d reconstruction of a human ear from a point cloud
WO2022156626A1 (zh) 一种图像的视线矫正方法、装置、电子设备、计算机可读存储介质及计算机程序产品
CN107622480B (zh) 一种Kinect深度图像增强方法
JP2018536915A (ja) ３次元再構成において構造特徴を検出し、組み合わせるための方法およびシステム
CN111243093A (zh) 三维人脸网格的生成方法、装置、设备及存储介质
WO2012153447A1 (ja) 画像処理装置、映像処理方法、プログラム、集積回路
CN106981078B (zh) 视线校正方法、装置、智能会议终端及存储介质
JP2009525543A (ja) 二次元画像からの三次元の顔の復元
KR102551713B1 (ko) 전자 장치 및 그 이미지 처리 방법
WO2015188666A1 (zh) 三维视频滤波方法和装置
CN107358609B (zh) 一种用于增强现实的图像叠加方法及装置
US20100079453A1 (en) 3D Depth Generation by Vanishing Line Detection
Fickel et al. Stereo matching and view interpolation based on image domain triangulation
IL257304A (en) Conversion of 2D to 3D video images
WO2022126674A1 (zh) 立体全景图像的质量评价方法、系统
Vu et al. Efficient hybrid tree-based stereo matching with applications to postcapture image refocusing
US20230394833A1 (en) Method, system and computer readable media for object detection coverage estimation
US20230024396A1 (en) A method for capturing and displaying a video stream
JP2013185905A (ja) 情報処理装置及び方法、並びにプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17845284

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17845284

Country of ref document: EP

Kind code of ref document: A1