WO2024055683A1 - 一种增强现实方法和计算装置 - Google Patents

一种增强现实方法和计算装置 Download PDF

Info

Publication number
WO2024055683A1
WO2024055683A1 PCT/CN2023/103178 CN2023103178W WO2024055683A1 WO 2024055683 A1 WO2024055683 A1 WO 2024055683A1 CN 2023103178 W CN2023103178 W CN 2023103178W WO 2024055683 A1 WO2024055683 A1 WO 2024055683A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinates
camera
image frame
point
pose data
Prior art date
Application number
PCT/CN2023/103178
Other languages
English (en)
French (fr)
Inventor
王骞鹏
林涛
轩文龙
时天欣
冯思淇
王润之
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2024055683A1 publication Critical patent/WO2024055683A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]

Definitions

  • the present application relates to the field of image processing, and in particular, to an augmented reality method and computing device.
  • Augmented reality is a technology that overlays virtual digital content with the real world.
  • An existing augmented reality method is roughly as follows: generate a point cloud of the surrounding background based on the positioning system, generate a plane based on the fitting point cloud, click on the intersection of the screen and the plane, calculate the intersection point of the screen and the plane as an anchor point, and use the anchor Click to project digital content to the camera and overlay the projected digital content with the background.
  • the present application provides an augmented reality method and computing device that can generate a surface of a closed space around the camera.
  • Virtual anchor points based on the surface can project digital content in various scenes. Since this method does not require background points The cloud generates anchor points and therefore has better applicability.
  • a first aspect provides an augmented reality method.
  • the method includes: using a camera to capture a first image frame and obtaining first camera pose data corresponding to the first image frame, determining a first tracking of the digital content in the first image frame. point coordinates and generate a first closed space including the optical center of the first camera, taking the ray emitted from the optical center coordinate of the first camera in the first camera posture and passing through the first tracking point coordinate as the first ray, and then determining the first virtual
  • the anchor point is the intersection point of the surface of the first closed space and the first ray.
  • the digital content is projected on the first image frame according to the first camera pose data and the coordinates of the first virtual anchor point to obtain an augmented reality image frame.
  • the first camera pose data includes the first camera optical center coordinates and the first camera pose data.
  • the shape of the first closed space may be, but is not limited to, a cube, a cylinder, or a cone.
  • the size of the first enclosed space is not limited.
  • a closed space can be generated according to actual needs, and the intersection between the ray emitted from the camera's optical center coordinates and the surface of the closed space serves as the virtual anchor point of the digital content.
  • virtual anchor points can be generated in any scene to realize the augmented reality function without relying on the background point cloud, so it has better applicability.
  • the above augmented reality method further includes: determining an image tracking area including the coordinates of the first tracking point in the first image frame; using a camera to capture the second image frame and obtaining the second The second camera pose data corresponding to the image frame; determine the second tracking point coordinates in the second image frame according to the image tracking area; according to the first camera pose data, the first tracking point coordinates, the second camera pose data and The second tracking point coordinates calculate the background point coordinates; when the first virtual camera pose corresponding to the second image frame is determined based on the first camera optical center coordinates, the coordinates of the first virtual anchor point, the background point coordinates and the second camera pose data.
  • the digital content is projected on the second image frame according to the first virtual camera pose data corresponding to the second image frame and the coordinates of the first virtual anchor point.
  • the second image frame may be any image frame after the first image frame.
  • the second camera pose data includes the second camera optical center coordinates and the second camera pose data.
  • Determining the first virtual camera pose data corresponding to the second image frame from the pose data includes: determining the first distance based on the optical center coordinates of the first camera and the coordinates of the first virtual anchor point; determining the first distance based on the optical center coordinates of the first camera and the background point coordinates.
  • the optical center coordinates of the virtual camera and the second camera posture data determine the first virtual camera posture data corresponding to the second image frame. Based on this, the scale alignment ratio can be determined, and the virtual camera optical center coordinates and virtual camera pose data calculated based on the scale alignment ratio can be projected to keep the digital content consistent with the background area.
  • the above augmented reality method also includes: when the digital content does not fit the background area of the second image frame, determining the second ray according to the second tracking point coordinates and the first virtual camera pose data corresponding to the second image frame; determining the second virtual anchor The point is the intersection point of the surface of the first closed space and the second ray; the digital content is projected on the second image frame according to the first virtual camera pose data corresponding to the second image frame and the coordinates of the second virtual anchor point.
  • the error in the background point coordinates may cause the digital content to be inconsistent with the background area.
  • the virtual camera pose and virtual anchor point can be updated without calculating the background point coordinates to implement the augmented reality function, thereby reducing the number of digital content and the background area.
  • the background does not fit well.
  • the above-mentioned augmented reality method further includes: when based on the first camera pose data, the first tracking point coordinates, the second camera pose data and the second When the coordinates of the tracking point cannot determine the coordinates of the background point, the optical center coordinates of the second virtual camera are determined based on the preset scale alignment ratio, the optical center coordinates of the first camera and the optical center coordinates of the second camera; the optical center coordinates of the second virtual camera and the optical center coordinates of the second virtual camera are determined.
  • the second camera pose data determines the second virtual camera pose data corresponding to the second image frame; determines the third ray according to the second virtual camera pose data corresponding to the second image frame and the second tracking point coordinates; determines the third virtual anchor point is the intersection point of the surface of the first closed space and the third ray; the digital content is projected on the second image frame according to the second virtual camera pose corresponding to the second image frame and the coordinates of the third virtual anchor point.
  • a method of calculating virtual camera pose data and virtual anchor points is provided to implement the augmented reality function and improve the flexibility of the solution implementation.
  • determining the first tracking point coordinates of the digital content in the first image frame includes: according to the The touch operation determines the coordinates of the touch point; the coordinates of the touch point are converted into coordinates of the first tracking point of the digital content in the first image frame. This allows you to touch anywhere on the screen and then determine the tracking point coordinates of the digital content based on that location.
  • determining the first tracking point coordinates of the digital content in the first image frame includes: according to the The touch operation determines that the coordinates of the first tracking point of the digital content in the first image frame are preset coordinates. This allows you to touch the screen to select preset coordinates as tracking point coordinates for digital content.
  • the above-mentioned augmented reality method further includes: using a camera to capture a third image frame and obtaining a corresponding image of the third image frame.
  • the third camera pose data is then used to determine the third tracking point coordinates of the digital content in the third image frame and generate a second closed space including the third camera optical center;
  • the ray emitted by the coordinates and passing through the coordinates of the third tracking point is regarded as the fourth ray, and the fourth virtual anchor point is determined as the intersection point of the second closed space and the fourth ray.
  • the third camera pose data includes the third camera optical center coordinates and the third camera pose data. This provides a method to update the virtual anchor point, based on which the position of the digital content can be adjusted.
  • the second aspect provides a computing device, which includes a camera, a pose sensor and a processor; the camera is used to capture a first image frame; the pose sensor is used to obtain the first camera pose data corresponding to the first image frame.
  • the camera pose data includes first camera optical center coordinates and first camera pose data; the processor is configured to determine the first tracking point coordinates of the digital content in the first image frame; generate a first closed space including the first camera optical center; After determining that the first ray is a ray that emanates from the optical center coordinates of the first camera in the first camera posture and passes through the coordinates of the first tracking point, the first virtual anchor point is determined to be the intersection point of the surface of the first closed space and the first ray, Project the digital content on the first image frame according to the first camera pose data and the coordinates of the first virtual anchor point to obtain an augmented reality image frame.
  • the processor is further configured to determine an image tracking area including the coordinates of the first tracking point in the first image frame; the camera is also configured to capture the second image frame; the pose sensor is also used to obtain the second camera pose data corresponding to the second image frame; the processor is also used to determine the coordinates of the second tracking point in the second image frame according to the image tracking area; when according to the first camera When determining the background point coordinates based on the pose data, the first tracking point coordinates, the second camera pose data and the second tracking point coordinates, the coordinates of the first camera optical center, the coordinates of the first virtual anchor point, the background point coordinates and the second The camera pose data determines the first virtual camera pose data corresponding to the second image frame; projecting the digital content on the second image frame according to the first virtual camera pose data corresponding to the second image frame and the coordinates of the first virtual anchor point superior.
  • the processor is specifically configured to determine the first camera optical center coordinates and the coordinates of the first virtual anchor point. distance; determine the second distance based on the coordinates of the optical center of the first camera and the coordinates of the background point; determine the scale alignment ratio as the ratio of the first distance to the second distance; determine the scale alignment ratio and the optical center of the second camera based on the coordinates of the optical center of the first camera The coordinates calculate the optical center coordinates of the first virtual camera; determine the first virtual camera pose data corresponding to the second image frame based on the optical center coordinates of the first virtual camera and the second camera pose data.
  • the processor is also used to When the digital content does not fit into the background area of the second image frame, determine the coordinates of the second tracking point in the second image frame according to the image tracking area; and determine the coordinates of the second tracking point and the first virtual camera position corresponding to the second image frame.
  • Digital content is projected onto the second image frame.
  • the processor is further used to determine the optical center coordinates of the second virtual camera according to a preset scale alignment ratio, the optical center coordinates of the first camera and the second camera when the background point coordinates cannot be determined according to the first camera pose data, the first tracking point coordinates, the second camera pose data and the second tracking point coordinates; determine the second virtual camera pose data corresponding to the second image frame according to the optical center coordinates of the second virtual camera and the second camera pose data; determine the third ray according to the second virtual camera pose data corresponding to the second image frame and the second tracking point coordinates; determine the third virtual anchor point as the intersection of the surface of the first closed space and the third ray; and project the digital content onto the second image frame according to the second virtual camera pose corresponding to the second image frame and the coordinates of the third virtual anchor point.
  • the processor is specifically configured to determine the coordinates of the touch point according to the touch operation acting on the screen; The point coordinates are converted to first tracking point coordinates of the digital content in the first image frame.
  • the processor is specifically configured to determine the digital content in the first image frame according to the touch operation acting on the screen.
  • the coordinates of the first tracking point are preset coordinates.
  • the camera is also used to capture a third image frame; the pose sensor is also used to obtain the corresponding image frame of the third image.
  • the third camera pose data includes the third camera optical center coordinates and the third camera pose data; the processor is also used to determine the third tracking point coordinates of the digital content in the third image frame; generate The second closed space including the optical center of the third camera; determine the fourth virtual anchor point as the intersection point of the second closed space and the fourth ray.
  • the fourth ray is emitted from and passes through the coordinates of the optical center of the third camera in the third camera attitude.
  • the ray of the coordinates of the third tracking point please refer to the corresponding records in the first aspect.
  • the third aspect provides a computing device, which includes a processor and a memory, the memory is used to store a program; the processor executes the program to implement the method described in the first aspect or any possible implementation of the first aspect. .
  • the fourth aspect provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are run on a computer, they cause the computer to execute the method described in the first aspect or any of the possible implementations of the first aspect.
  • the fifth aspect provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the method described in the first aspect or any possible implementation of the first aspect.
  • the sixth aspect provides a chip system.
  • the chip system includes at least one processor and an interface.
  • the interface is used to receive data and/or instructions.
  • the at least one processor is used to support a computing device to implement the first aspect or any of the first aspects.
  • One possible implementation involves the functionality.
  • the chip system may also include a memory for storing necessary program instructions and data of the computing device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • Figure 1A is a schematic diagram of an indoor scene in an embodiment of the present application.
  • Figure 1B is a schematic diagram of digital content of an indoor scene in an embodiment of the present application.
  • Figure 2 is a flow chart of the augmented reality method in the embodiment of the present application.
  • Figure 3 is a schematic diagram of selecting tracking point coordinates and image tracking areas in the embodiment of the present application.
  • Figure 4 is a schematic diagram of a closed space and a virtual anchor point in the embodiment of the present application.
  • Figure 5 is another flow chart of the augmented reality method in the embodiment of the present application.
  • Figure 6 is a schematic diagram of generating virtual camera poses and virtual anchor points in the embodiment of the present application.
  • Figure 7 is another schematic diagram of generating a virtual camera pose and a virtual anchor point in an embodiment of the present application.
  • Figure 8 is another schematic diagram of generating a virtual camera pose and a virtual anchor point in an embodiment of the present application.
  • Figure 9 is a schematic diagram of correcting tracking point coordinates in the embodiment of the present application.
  • Figure 10 is a structural diagram of a computing device in an embodiment of the present application.
  • Figure 11 is another structural diagram of a computing device in an embodiment of the present application.
  • the augmented reality method of the present application can be applied to computing devices.
  • the computing device may be a terminal or a chip, and the terminal may also be called a terminal device, user equipment (UE), mobile station, mobile terminal, etc.
  • Terminals can be widely used in various scenarios, such as device-to-device (D2D), vehicle to everything (V2X) communication, machine-type communication (MTC), Internet of Things ( internet of things (IOT), virtual reality, augmented reality, industrial control, autonomous driving, telemedicine, smart grid, smart furniture, smart office, smart wear, smart transportation, smart city, etc.
  • D2D device-to-device
  • V2X vehicle to everything
  • MTC machine-type communication
  • IOT Internet of Things
  • virtual reality augmented reality
  • industrial control autonomous driving
  • telemedicine smart grid
  • smart furniture smart office
  • smart wear smart transportation
  • smart city etc.
  • the terminal can be a mobile phone, tablet computer, AR device, virtual reality (VR) device, mixed reality (MR) device, computer with wireless transceiver function, wearable device, vehicle, airplane, ship, robot, Robotic arms, smart home devices, etc.
  • AR device virtual reality
  • VR virtual reality
  • MR mixed reality
  • the embodiments of this application do not limit the specific technology and specific equipment form used by the terminal.
  • FIG. 1A and FIG. 1B when the user takes a view through the camera, an indoor scene image as shown in FIG. 1A is captured. Users can select digital content on the screen (such as a cartoon squirrel) to place on the ground, as shown in Figure 1B.
  • digital content on the screen such as a cartoon squirrel
  • the augmented reality method of the present application includes:
  • Step 201 Use a camera to capture a first image frame and obtain first camera pose data corresponding to the first image frame.
  • any image frame can be selected as the first image frame.
  • the camera pose data can be determined by the pose sensor.
  • the pose sensor may include one or more of image sensing data, depth data, inertial sensor data, magnetometer data, and a positioning unit.
  • Inertial sensor data is measured by an inertial measurement unit (IMU).
  • the positioning unit can be a simultaneous localization and mapping (SLAM) unit, a visual-inertial odometry (VIO) unit or other positioning units.
  • the positioning unit may also include a back-end bundle adjustment module (BA) optimization module or a closed-loop correction module.
  • BA back-end bundle adjustment module
  • the positioning unit determines the camera pose data based on one or more of image sensing data, depth data, inertial sensor data, and magnetometer data.
  • the camera pose data can be 6 degrees of freedom (degree-o-freedom, DOF) data or 3DOF data.
  • 6DOF data includes three-dimensional coordinates and attitude data.
  • 3DOF includes three-dimensional coordinates or attitude data.
  • the three-dimensional coordinates include (X-axis coordinates, Y-axis coordinates, and Z-axis coordinates), and the attitude data includes (roll angle, pitch angle, and tilt angle).
  • the first camera pose data includes the first camera optical center coordinates and the first camera pose data, and the first camera pose data is used to represent the first camera pose.
  • Step 202 Determine the coordinates of the first tracking point of the digital content in the first image frame.
  • the first tracking point coordinates correspond to the position of the digital content in the scene, which is also called a background point.
  • Digital content includes text, images, sounds, three-dimensional objects, etc. that exist in digital form.
  • the computing device may determine touch point coordinates according to a touch operation on the screen; convert the touch point coordinates into first tracking point coordinates of the digital content in the first image frame.
  • the computing device determines the coordinates of the first tracking point of the digital content in the first image frame as the preset coordinates according to the touch operation on the screen.
  • the touch operation may be, but is not limited to, a click operation or a drag operation.
  • an image tracking area including the coordinates of the first tracking point may also be generated.
  • the image tracking area may be an area centered on the coordinates of the first tracking point.
  • the image tracking areas of different image frames correspond to the same background area.
  • the image tracking area can be rectangular, square or circular, or other shapes.
  • the first tracking point coordinates 31 and the image tracking area 32 are generated according to the click position.
  • Step 203 Generate a first closed space including the optical center of the first camera.
  • a closed space can be generated based on the closed space parameters.
  • Enclosed space parameter values can be preset or entered by the user.
  • Closed space parameters include shape type and space size parameters.
  • the shape type can be cube, cube, cylinder or cone, etc.
  • Each shape can be configured with corresponding spatial size parameters.
  • the spatial size parameters of a cube include length, width and height.
  • the spatial dimension parameters of the cube include the side length.
  • the spatial parameters of a cylinder include base radius and height.
  • the closed space shape and space size parameters can be selected according to the actual situation, and are not limited in this application.
  • Step 204 Determine the first virtual anchor point as the intersection point of the surface of the first closed space and the first ray.
  • the first ray is a ray emitted from the optical center coordinates of the first camera and passing through the coordinates of the first tracking point in the first camera attitude.
  • Figure 4 is a schematic diagram of the enclosed space and anchor points. Referring to Figure 4, in one example, the closed space 41 is a cube. After the ray emitted from the camera optical center 401 passes through the tracking point coordinate 402, it intersects with the surface of the closed space 41 at the virtual anchor point 403. The background point in the background 42 421 is located on the extension of this ray.
  • Step 205 Project the digital content on the first image frame according to the first camera pose data and the coordinates of the first virtual anchor point to obtain an augmented reality image frame. It should be understood that steps such as rendering can be performed after projection, which is not limited in this application. Step 202 and step 203 have no fixed order.
  • a closed space can be generated according to actual needs.
  • the intersection between the ray emitted from the optical center coordinates of the first camera and the surface of the first closed space serves as the first virtual anchor point of the digital content.
  • the data and the first virtual anchor point can project digital content to the first image frame, thereby obtaining an augmented reality image.
  • This does not rely on the background point cloud to generate anchor points, and digital content can be placed at any location when there is no environmental data. It can also fit the digital content to the background, so it has better applicability.
  • the augmented reality method of the present application also includes:
  • Step 501 Determine the image tracking area including the coordinates of the first tracking point in the first image frame.
  • Step 502 Use the camera to capture a second image frame and obtain second camera pose data corresponding to the second image frame.
  • the second image frame may be any image frame after the first image frame.
  • the second camera pose data includes the second camera optical center coordinates and the second camera pose data, and the second camera pose data corresponds to the second camera pose.
  • Step 503 Determine the coordinates of the second tracking point in the second image frame according to the image tracking area.
  • the image tracking area of the second image frame may be determined based on the image tracking area of the first image frame, and then the coordinates of the second tracking point are determined to be the center coordinates of the image tracking area of the second image frame.
  • Step 504 Determine whether the background point coordinates can be determined based on the first camera pose data, the first tracking point coordinates, the second camera pose data and the second tracking point coordinates. If yes, execute step 505. If not, execute step 507.
  • Step 505 Determine the first virtual camera pose data corresponding to the second image frame according to the first camera optical center coordinates, the coordinates of the first virtual anchor point, the background point coordinates and the second camera pose data.
  • the above data are processed using the triangulation method in computer vision positioning technology to obtain the background points. coordinate.
  • step 505 includes: determining the first distance according to the coordinates of the optical center of the first camera and the coordinates of the first virtual anchor point; determining the second distance according to the coordinates of the optical center of the first camera and the coordinates of the background point; determining the scale alignment ratio to be the first distance.
  • the ratio of the first distance to the second distance calculating the optical center coordinates of the first virtual camera based on the optical center coordinates of the first camera, the scale alignment ratio and the optical center coordinates of the second camera; calculating the optical center coordinates of the first virtual camera based on the optical center coordinates of the first virtual camera and the attitude data of the second camera Determine the first virtual camera pose data corresponding to the second image frame.
  • the first virtual camera pose data corresponding to the second image frame includes the first virtual camera optical center coordinates and the second camera pose data.
  • the first distance is denoted as Dv
  • the second distance is denoted as Dt
  • the scale alignment ratio is marked as Scale
  • the first camera optical center coordinate is marked as refPosition
  • the second camera optical center coordinate is marked as currPosition
  • the virtual camera optical center coordinate is marked as virtualPosition.
  • the closed space 61 is a cube
  • the distance from the camera optical center 601 to the virtual anchor point 603 is Dv
  • the distance from the camera optical center 601 to the background point 621 is Dt.
  • the coordinates of the camera optical center 601 are refPosition, and the coordinates of the camera optical center 631 are currPosition.
  • virtualPosition Scale*(currPosition-refPosition)
  • the coordinates of the virtual camera optical center 632 can be calculated as virtualPosition.
  • the virtual camera posture data corresponding to the virtual camera optical center 632 is the same as the camera posture data corresponding to the camera optical center 631 .
  • the scaling ratio of the digital content is the same as that of the background during projection. , so that the digital content fits the background.
  • Step 506 Project the digital content on the second image frame according to the first virtual camera pose data corresponding to the second image frame and the coordinates of the first virtual anchor point.
  • Step 507 Determine the optical center coordinates of the second virtual camera according to the preset scale alignment ratio, the optical center coordinates of the first camera and the optical center coordinates of the second camera.
  • the preset scale alignment ratio Sc can be 0.
  • this application may not perform step 507 and directly set the optical center coordinates of the second virtual camera to the preset optical center coordinates of the virtual camera.
  • the preset optical center coordinates of the virtual camera may be set according to the actual situation. There are no restrictions on application.
  • Step 508 Determine the second virtual camera pose data corresponding to the second image frame according to the second virtual camera optical center coordinates and the second camera pose data.
  • the second virtual camera pose data includes the optical center coordinates of the second virtual camera and the second camera pose data.
  • Step 509 Determine a third ray according to the second virtual camera posture data corresponding to the second image frame and the second tracking point coordinates.
  • the third ray is a ray emitted from the optical center coordinates of the second virtual camera under the second camera posture and passing through the second tracking point coordinates.
  • Step 510 Determine the third virtual anchor point as the intersection point of the surface of the first closed space and the third ray.
  • Step 511 Project the digital content on the second image frame according to the second virtual camera pose corresponding to the second image frame and the coordinates of the third virtual anchor point.
  • the closed space 71 is a cube
  • the camera optical center 701 the virtual anchor point 703 and the background point 721 are in a line. in a straight line.
  • the camera pose data at the camera optical center 731 can be obtained, and then the virtual camera pose can be determined based on the camera pose data corresponding to the camera optical center 731 and the coordinates of the virtual camera optical center 741.
  • the ray 743 is determined based on the virtual camera optical center 741 and the tracking point 742, and the intersection point of the ray 743 and the closed space 71 is used as the virtual anchor point 744.
  • the digital content is then projected according to the virtual camera pose and the coordinates of the virtual anchor point 744 to obtain an augmented reality image frame.
  • step 501 may be executed after step 202.
  • Step 502 is performed after step 205.
  • image tracking area please refer to the corresponding description in the embodiment shown in FIG. 2 .
  • the first virtual camera pose and scale alignment ratio corresponding to the second image frame can be determined.
  • the first virtual camera pose corresponding to the second image frame can be determined.
  • the scale alignment ratio can realize the augmented reality function, and can scale the digital content according to the scale alignment ratio, so that the digital content fits the background.
  • the second virtual camera pose and the coordinates of the third virtual anchor point can be obtained, thereby realizing the augmented reality function, thereby improving the flexibility of the solution implementation.
  • the third virtual anchor point and the first virtual anchor point are on the same plane of the first closed space, the scaling ratio of the digital content and the background can be consistent, so that the digital content and the background fit together.
  • the augmented reality method of this application also includes: when the digital content does not fit the background area of the second image frame, according to The second tracking point coordinates and the first virtual camera pose data corresponding to the second image frame determine the second ray; determine the second virtual anchor point as the intersection point of the surface of the first closed space and the second ray; correspond to the second image frame The first virtual camera pose data and the coordinates of the second virtual anchor point project the digital content onto the second image frame.
  • the second ray is a ray emitted from the optical center coordinates of the first virtual camera and passing through the second tracking point coordinates in the second camera posture.
  • the first virtual camera posture data and the second tracking point coordinates corresponding to the second image frame can be used to re-determine the virtual anchor point to obtain the target Augmented reality image frames.
  • This application may not display the augmented reality image frame generated in step 506 and display the target augmented reality image frame, which can reduce image frames whose data content does not fit the background.
  • the positions of the camera center 801 , the virtual anchor point 803 and the virtual camera optical center 832 are as shown in FIG. 8 .
  • the projection point is the projection point 912 shown in Figure 9.
  • Projection point 912 is not the center point of image tracking area 92, so the cartoon squirrel cannot fit into the background.
  • the augmented reality method of the present application also includes: using a camera to capture a third image frame and obtaining the third camera pose data corresponding to the third image frame.
  • the coordinates of the third tracking point of the digital content are determined in the frame; the fourth ray is determined to be a ray emitted from the coordinates of the optical center of the third camera in the third camera posture and passing through the coordinates of the third tracking point, and a third ray including the optical center of the third camera is generated.
  • the third camera pose data includes the third camera optical center coordinates and the third camera pose data, and the third camera pose data corresponds to the third camera pose.
  • the closed space, virtual anchor point and virtual camera optical center can be updated, and then the augmented reality function can be implemented based on the updated virtual anchor point and the updated virtual camera optical center.
  • the specific process of generating a closed space and calculating virtual anchor points can be found in the corresponding records of the embodiment shown in Figure 2, and will not be described again here.
  • a computing device 1000 capable of implementing the augmented reality method in the above embodiment.
  • a computing device 1000 includes a camera 1001 , a pose sensor 1002 and a processor 1003 .
  • the camera 1001 is used to capture the first image frame; the pose sensor 1002 is used to obtain the first camera pose data corresponding to the first image frame, and the processor 1003 is used to determine the first tracking point coordinates of the digital content in the first image frame ; Generate the first closed space including the optical center of the first camera; use the ray emitted from the optical center coordinates of the first camera in the first camera posture and passing through the coordinates of the first tracking point as the first ray, and determine the first virtual anchor point as The intersection point of the surface of the first closed space and the first ray projects the digital content on the first image frame according to the first camera pose data and the coordinates of the first virtual anchor point to obtain an augmented reality image frame.
  • the processor 1003 is also used to determine the image tracking area including the coordinates of the first tracking point in the first image frame; the camera 1001 is also used to capture the second image frame; the pose sensor 1002 is also used to Obtain the second camera pose data corresponding to the second image frame; the processor 1003 is also configured to determine the coordinates of the second tracking point in the second image frame according to the image tracking area; when based on the first camera pose data, the first tracking When determining the background point coordinates, the second camera pose data and the second tracking point coordinates, the second point coordinates are determined based on the first camera optical center coordinates, the first virtual anchor point coordinates, the background point coordinates and the second camera pose data.
  • the first virtual camera pose data corresponding to the image frame project the digital content on the second image frame according to the first virtual camera pose data corresponding to the second image frame and the coordinates of the first virtual anchor point.
  • the processor 1003 is specifically configured to determine the first distance according to the coordinates of the optical center of the first camera and the coordinates of the first virtual anchor point; and to determine the second distance according to the coordinates of the optical center of the first camera and the coordinates of the background point. ; Determine the scale alignment ratio as the ratio of the first distance to the second distance; Calculate the first virtual camera optical center coordinates according to the first camera optical center coordinates, the scale alignment ratio and the second camera optical center coordinates; According to the first virtual camera optical center coordinates The coordinates and the second camera pose data determine the first virtual camera pose data corresponding to the second image frame.
  • the processor 1003 is also configured to, when the digital content does not fit the background area of the second image frame, calculate the first virtual camera pose corresponding to the second tracking point coordinates and the second image frame.
  • the data determines the second ray; determines the second virtual anchor point as the intersection point of the surface of the first closed space and the second ray; and converts the number according to the first virtual camera pose data corresponding to the second image frame and the coordinates of the second virtual anchor point.
  • the content is projected on the second image frame.
  • the processor 1003 is also configured to determine the background point coordinates according to the first camera pose data, the first tracking point coordinates, the second camera pose data and the second tracking point coordinates.
  • the preset scale alignment ratio, the first camera optical center coordinates and the second camera optical center coordinates determine the second virtual camera optical center coordinates; determine the second image frame corresponding to the second virtual camera optical center coordinates and the second camera attitude data.
  • second virtual camera pose data determine the third ray according to the second virtual camera pose data corresponding to the second image frame and the coordinates of the second tracking point; determine the third virtual anchor point as the surface of the first closed space and the third ray the intersection point; project the digital content on the second image frame according to the second virtual camera pose corresponding to the second image frame and the coordinates of the third virtual anchor point.
  • the processor 1003 is specifically configured to determine the coordinates of the touch point according to the touch operation on the screen; and convert the coordinates of the touch point into the coordinates of the first tracking point of the digital content in the first image frame.
  • the processor 1003 is specifically configured to determine the coordinates of the first tracking point of the digital content in the first image frame as preset coordinates according to the touch operation on the screen.
  • the camera 1001 is also used to capture a third image frame; the pose sensor 1002 is also used to acquire the third camera pose data corresponding to the third image frame, and the third camera pose data includes a third Camera optical center coordinates and third camera attitude data; processor 1003 is also used to determine the third tracking point coordinates of the digital content in the third image frame; generate a second closed space including the optical center of the third camera; determine that the fourth ray is from the optical center of the third camera under the third camera attitude. The ray emitted by the coordinates and passing through the coordinates of the third tracking point determines the fourth virtual anchor point as the intersection point of the second closed space and the fourth ray.
  • the steps performed by the pose sensor 1002 can be implemented by the camera 1001.
  • the steps and beneficial effects performed by the camera 1001, the posture sensor 1002 and the processor 1003, please refer to the corresponding records in the embodiment shown in Figure 2, the embodiment shown in Figure 5 or the optional embodiment.
  • the present application also provides a computing device 1100 capable of implementing the enhanced implementation method in the above embodiment.
  • the computing device 1100 includes: a camera 1101 , a depth sensor 1102 , a processor 1103 , an inertial measurement unit 1104 , an input unit 1105 , a display unit 1106 , a memory 1107 , a communication interface 1108 and a magnetometer 1109 .
  • the camera 1101, depth sensor 1102, inertial measurement unit 1104, input unit 1105, display unit 1106, memory 1107, communication interface 1108 and magnetometer 1109 are connected to the processor 1103 through a bus.
  • the bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus can be divided into address bus, data bus, control bus, etc.
  • a bus may include a path that carries information between components of computing device 1100 .
  • the processor 1103 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). any one or more of them.
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • the depth sensor 1102 is used to measure depth information of the scene.
  • Depth sensor 1102 may be, but is not limited to, a laser sensor.
  • the inertial measurement unit 1104 is used to obtain motion data of the computing device, such as linear acceleration, angular velocity, etc.
  • the input unit 1105 may be a touch screen, keyboard, mouse or other input device.
  • the display unit 1106 may be a display.
  • the magnetometer 1109 is used to measure the yaw angle of the computing device.
  • Memory 1107 may include volatile memory, such as random access memory (RAM).
  • RAM random access memory
  • the processor 104 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, mechanical hard disk drive (hard disk drive, HDD) or solid state drive (solid state drive). drive, SSD).
  • ROM read-only memory
  • HDD hard disk drive
  • SSD solid state drive
  • the memory 1107 stores executable program code
  • the processor 1103 executes the executable program code to implement the functions of the aforementioned camera, pose sensor, and processor respectively, thereby implementing the augmented reality method. That is, the memory 1107 stores instructions for executing the augmented reality method.
  • the communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to implement communication between the computing device 1100 and other devices or communication networks.
  • this application does not limit the number of cameras, depth sensors, processors, inertial measurement units, input units, display units, memories, communication interfaces and magnetometers in the computing device 1100 .
  • An embodiment of the present application also provides a computer program product containing instructions.
  • the computer program product may be a software or program product containing instructions capable of running on a computing device or stored in any available medium.
  • the computer program product when run on at least one computing device, causes at least one computing device to perform the augmented reality method.
  • An embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium may be any available medium that a computing device can store, or a data storage device such as a data center that contains one or more available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, tape), optical media (eg, DVD), or semiconductor media (eg, solid state drive), etc.
  • the computer-readable storage medium includes instructions that instruct a computing device to perform an augmented reality method.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种增强现实方法包括：摄取第一图像帧以及获取第一相机位姿数据后，在第一图像帧中确定数字内容的第一跟踪点坐标以及生成包括第一相机光心坐标的第一封闭空间，将第一射线和第一封闭空间的表面的交点作为第一虚拟锚点，根据第一相机光心坐标和第一虚拟锚点的坐标将数字内容投影在第一图像帧上，以得到增强现实图像帧；后续图像帧通过计算图像帧对应的虚拟位姿，可以将数字内容渲染到对应的背景纹理上，且数字内容与背景的大小变化保持一致。这样可以在任何场景生成虚拟锚点以实现增强现实功能，具有更好的适用性。本申请还提供一种能够实现上述增强现实方法的计算装置。

Description

一种增强现实方法和计算装置
本申请要求于2022年09月16日提交中国专利局、申请号为202211130875.4、申请名称为“一种增强现实方法和计算装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理领域，尤其涉及一种增强现实方法和计算装置。
背景技术
增强现实(augmented reality，AR)是一种将虚拟数字内容与真实世界叠加的技术。
现有一种增强现实的方法大致如下：根据定位系统生成周围背景的点云，根据拟合点云生成平面，点击屏幕与平面的交汇处，计算屏幕与平面的交汇点作为锚点，根据该锚点将数字内容投影至相机，将投影后的数字内容与背景叠加。
对于不能生成背景点云的场景，上述方法不能生成锚点，也就无法实现上述方法。
发明内容
有鉴于此，本申请提供一种增强现实方法和计算装置，能够在相机周围生成封闭空间的表面，基于该表面的虚拟锚点能够将数字内容投影在各种场景，由于该方法不需要背景点云生成锚点，因此具有更好的适用性。
第一方面提供一种增强现实方法，该方法包括：使用相机摄取第一图像帧以及获取第一图像帧对应的第一相机位姿数据后，在第一图像帧中确定数字内容的第一跟踪点坐标以及生成包括第一相机光心的第一封闭空间，将在第一相机姿态下从第一相机光心坐标发出且经过第一跟踪点坐标的射线作为第一射线，再确定第一虚拟锚点为第一封闭空间的表面与第一射线的交点，根据第一相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第一图像帧上，以得到增强现实图像帧。其中，第一相机位姿数据包括第一相机光心坐标和第一相机姿态数据。第一封闭空间的形状可以是但不限于立方体，圆柱体，圆锥体。第一封闭空间的大小不作限定。
依此实施，能够根据实际需要生成封闭空间，从相机光心坐标发出的射线与封闭空间的表面之间的交点作为数字内容的虚拟锚点。这样不依赖背景点云生成锚点，可以在任何场景生成虚拟锚点以实现增强现实功能，因此具有更好的适用性。
在第一方面的第一种可能的实现方式中，上述增强现实方法还包括：在第一图像帧中确定包括第一跟踪点坐标的图像跟踪区域；使用相机摄取第二图像帧以及获取第二图像帧对应的第二相机位姿数据；根据图像跟踪区域确定在第二图像帧中的第二跟踪点坐标；根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标计算背景点坐标；当根据第一相机光心坐标，第一虚拟锚点的坐标，背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据时，根据第二图像帧对应的第一虚拟相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第二图像帧上。第二图像帧可以是第一图像帧之后的任意一个图像帧。第二相机位姿数据包括第二相机光心坐标和第二相机姿态数据。依此可以实时计算第一图像帧之后的每个图像帧对应的虚拟相机位姿，根据该虚拟相机位姿和第一虚拟锚点可以投影得到每个图像帧对应的增强现实帧。
结合第一方面的第一种可能的实现方式，在第一方面的第二种可能的实现方式中，根据第一相机光心坐标，第一虚拟锚点的坐标，背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据包括：根据第一相机光心坐标和第一虚拟锚点的坐标确定第一距离；根据第一相机光心坐标和背景点坐标确定第二距离；确定尺度对齐比例为第一距离与第二距离之比；根据第一相机光心坐标，尺度对齐比例和第二相机光心坐标计算第一虚拟相机光心坐标；根据第一虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第一虚拟相机位姿数据。依此可以确定尺度对齐比例，根据尺度对齐比例计算出的虚拟相机光心坐标和虚拟相机位姿数据进行投影，能够将数字内容与背景区域保持贴合。
结合第一方面的第二种可能的实现方式，在第一方面的第三种可能的实现方式中，上述增强现实方 法还包括：当数字内容与第二图像帧的背景区域没有贴合时，根据第二跟踪点坐标和第二图像帧对应的第一虚拟相机位姿数据确定第二射线；确定第二虚拟锚点为第一封闭空间的表面与第二射线的交点；根据第二图像帧对应的第一虚拟相机位姿数据和第二虚拟锚点的坐标将数字内容投影在第二图像帧上。背景点坐标的误差可能导致数字内容与背景区域不贴合，依此实施能够在不计算背景点坐标的情况下更新虚拟相机位姿和虚拟锚点，以实现增强现实功能，从而减少数字内容与背景不贴合的情况。
结合第一方面，在第一方面的第四种可能的实现方式中，上述增强现实方法还包括：当根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标不能确定背景点坐标时，根据预设的尺度对齐比例、第一相机光心坐标和第二相机光心坐标确定第二虚拟相机光心坐标；根据第二虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第二虚拟相机位姿数据；根据第二图像帧对应的第二虚拟相机位姿数据和第二跟踪点坐标确定第三射线；确定第三虚拟锚点为第一封闭空间的表面与第三射线的交点；根据第二图像帧对应的第二虚拟相机位姿和第三虚拟锚点的坐标将数字内容投影在第二图像帧上。在不能计算出背景点坐标时，依此提供了一种计算虚拟相机位姿数据和虚拟锚点的方法，以实现增强现实功能，提高了方案实施的灵活性。
结合第一方面或第一方面以上任一实现方式，在第一方面的第五种可能的实现方式中，在第一图像帧中确定数字内容的第一跟踪点坐标包括：根据作用于屏幕的触控操作确定触控点坐标；将触控点坐标转换为在第一图像帧中数字内容的第一跟踪点坐标。这样可以触摸屏幕的任一位置，然后根据该位置确定数字内容的跟踪点坐标。
结合第一方面或第一方面以上任一实现方式，在第一方面的第六种可能的实现方式中，在第一图像帧中确定数字内容的第一跟踪点坐标包括：根据作用于屏幕的触控操作确定第一图像帧中数字内容的第一跟踪点坐标为预设坐标。这样可以触摸屏幕选择预设坐标作为数字内容的跟踪点坐标。
结合第一方面或第一方面以上任一实现方式，在第一方面的第七种可能的实现方式中，上述增强现实方法还包括：使用相机摄取第三图像帧以及获取第三图像帧对应的第三相机位姿数据，然后在第三图像帧中确定数字内容的第三跟踪点坐标以及生成包括第三相机光心的第二封闭空间；将在第三相机姿态下从第三相机光心坐标发出且经过第三跟踪点坐标的射线作为第四射线，确定第四虚拟锚点为第二封闭空间与第四射线的交点。第三相机位姿数据包括第三相机光心坐标和第三相机姿态数据。这样提供了一种更新虚拟锚点的方法，基于更新后的虚拟锚点可以调整数字内容的位置。
第二方面提供一种计算装置，其包括相机，位姿传感器和处理器；相机用于摄取第一图像帧；位姿传感器用于获取第一图像帧对应的第一相机位姿数据，第一相机位姿数据包括第一相机光心坐标和第一相机姿态数据；处理器用于在第一图像帧中确定数字内容的第一跟踪点坐标；生成包括第一相机光心的第一封闭空间；确定第一射线为在第一相机姿态下从第一相机光心坐标发出且经过第一跟踪点坐标的射线后，确定第一虚拟锚点为第一封闭空间的表面与第一射线的交点，根据第一相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第一图像帧上，以得到增强现实图像帧。
结合第二方面，在第二方面的第一种可能的实现方式中，处理器还用于在第一图像帧中确定包括第一跟踪点坐标的图像跟踪区域；相机还用于摄取第二图像帧；位姿传感器还用于获取第二图像帧对应的第二相机位姿数据；处理器还用于根据图像跟踪区域确定在第二图像帧中的第二跟踪点坐标；当根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标确定背景点坐标时，根据第一相机光心坐标，第一虚拟锚点的坐标，背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据；根据第二图像帧对应的第一虚拟相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第二图像帧上。
结合第二方面的第一种可能的实现方式，在第二方面的第二种可能的实现方式中，处理器具体用于根据第一相机光心坐标和第一虚拟锚点的坐标确定第一距离；根据第一相机光心坐标和背景点坐标确定第二距离；确定尺度对齐比例为第一距离与第二距离之比；根据第一相机光心坐标，尺度对齐比例和第二相机光心坐标计算第一虚拟相机光心坐标；根据第一虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第一虚拟相机位姿数据。
结合第二方面的第二种可能的实现方式，在第二方面的第三种可能的实现方式中，处理器还用于当 数字内容与第二图像帧的背景区域没有贴合时，根据图像跟踪区域在第二图像帧中确定第二跟踪点坐标；根据第二跟踪点坐标和第二图像帧对应的第一虚拟相机位姿数据确定第二射线；确定第二虚拟锚点为第一封闭空间的表面与第二射线的交点；根据第二图像帧对应的第一虚拟相机位姿数据和第二虚拟锚点的坐标将数字内容投影在第二图像帧上。
结合第二方面，在第二方面的第四种可能的实现方式中，处理器还用于当根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标不能确定背景点坐标时，根据预设的尺度对齐比例、第一相机光心坐标和第二相机光心坐标确定第二虚拟相机光心坐标；根据第二虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第二虚拟相机位姿数据；根据第二图像帧对应的第二虚拟相机位姿数据和第二跟踪点坐标确定第三射线；确定第三虚拟锚点为第一封闭空间的表面与第三射线的交点；根据第二图像帧对应的第二虚拟相机位姿和第三虚拟锚点的坐标将数字内容投影在第二图像帧上。
结合第二方面或第二方面以上任一实现方式，在第二方面的第五种可能的实现方式中，处理器具体用于根据作用于屏幕的触控操作确定触控点坐标；将触控点坐标转换为在第一图像帧中数字内容的第一跟踪点坐标。
结合第二方面或第二方面以上任一实现方式，在第二方面的第六种可能的实现方式中，处理器具体用于根据作用于屏幕的触控操作确定第一图像帧中数字内容的第一跟踪点坐标为预设坐标。
结合第二方面或第二方面以上任一实现方式，在第二方面的第七种可能的实现方式中，相机还用于摄取第三图像帧；位姿传感器还用于获取第三图像帧对应的第三相机位姿数据，第三相机位姿数据包括第三相机光心坐标和第三相机姿态数据；处理器还用于在第三图像帧中确定数字内容的第三跟踪点坐标；生成包括第三相机光心的第二封闭空间；确定第四虚拟锚点为第二封闭空间与第四射线的交点，第四射线是在第三相机姿态下从第三相机光心坐标发出且经过第三跟踪点坐标的射线。第二方面的名词解释，各单元执行的步骤和有益效果可参阅第一方面的相应记载。
第三方面提供一种计算装置，其包括处理器和存储器，所述存储器用于存储程序；所述处理器通过执行程序用于实现第一方面或第一方面任意一种可能实现方式记载的方法。
第四方面提供一种计算机可读存储介质，计算机可读存储介质中存储有指令，当指令在计算机上运行时，使得计算机执行第一方面或第一方面任意一种可能实现方式记载的方法。
第五方面提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行第一方面或第一方面任意一种可能实现方式记载的方法。
第六方面提供了一种芯片系统，该芯片系统包括至少一个处理器和接口，该接口用于接收数据和/或指令，至少一个处理器用于支持计算装置实现第一方面或第一方面的任意一种可能实现方式涉及的功能。在一种可能的设计中，芯片系统还可以包括存储器，存储器，用于保存计算装置必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包含芯片和其他分立器件。
附图说明
图1A为本申请实施例中室内场景的一个示意图；
图1B为本申请实施例中室内场景的数字内容的一个示意图；
图2为本申请实施例中增强现实方法的一个流程图；
图3为本申请实施例中选择跟踪点坐标和图像跟踪区域的一个示意图；
图4为本申请实施例中封闭空间和虚拟锚点的一个示意图；
图5为本申请实施例中增强现实方法的另一个流程图；
图6为本申请实施例中生成虚拟相机位姿和虚拟锚点的一个示意图；
图7为本申请实施例中生成虚拟相机位姿和虚拟锚点的另一个示意图；
图8为本申请实施例中生成虚拟相机位姿和虚拟锚点的另一个示意图；
图9为本申请实施例中纠正跟踪点坐标的一个示意图；
图10为本申请实施例中计算装置的一个结构图；
图11为本申请实施例中计算装置的另一个结构图。
具体实施方式
本申请的增强现实方法可以应用于计算装置。计算装置可以是终端或芯片，终端也可以称为终端设备、用户设备(user equipment，UE)、移动台、移动终端等。终端可以广泛应用于各种场景，例如，设备到设备(device-to-device，D2D)、车物(vehicle to everything，V2X)通信、机器类通信(machine-type communication，MTC)、物联网(internet of things，IOT)、虚拟现实、增强现实、工业控制、自动驾驶、远程医疗、智能电网、智能家具、智能办公、智能穿戴、智能交通、智慧城市等。
终端可以是手机、平板电脑、AR设备、虚拟现实(virtual reality，VR)设备、混合现实(mixed reality，MR)设备、带无线收发功能的电脑、可穿戴设备、车辆、飞机、轮船、机器人、机械臂、智能家居设备等。本申请的实施例对终端所采用的具体技术和具体设备形态不做限定。
参阅图1A和图1B，在一个实施例中，用户通过相机取景时，摄取如图1A所示的室内场景图像。用户可以在屏幕上选择数字内容(如卡通松鼠)放在地面上，如图1B所示。
有一些实际场景难以生成背景点云，按照现有方法就不能实现增强现实功能。本申请提供一种增强现实方法能够克服上述问题。下面对其进行介绍，参阅图2，在一个实施例中，本申请的增强现实方法包括：
步骤201、使用相机摄取第一图像帧以及获取第一图像帧对应的第一相机位姿数据。
本实施例中，在使用相机取景时，可以选择任意一张图像帧作为第一图像帧。相机位姿数据可以是位姿传感器确定的。位姿传感器可以包括图像传感数据，深度数据，惯性传感器数据，磁力计数据中的一项或多项以及定位单元。惯性传感器数据是通过惯性测量单元(inertial measurement unit，IMU)测量得到的。定位单元可以是即时定位与地图构建(simultaneous localization and mapping，SLAM)单元,视觉惯性里程计(visual-inertial odometry，VIO)单元或其他定位单元。定位单元还可以包括后端捆绑调整模块(bundle adjustment，BA)优化模块或闭环纠正模块。
具体的，定位单元根据图像传感数据，深度数据，惯性传感器数据，磁力计数据中的一项或多项确定相机位姿数据。相机位姿数据可以是6自由度(degree-o-freedom，DOF)数据或3DOF数据。6DOF数据包括三维坐标和姿态数据。3DOF包括三维坐标或姿态数据。三维坐标包括(X轴坐标，Y轴坐标，Z轴坐标)，姿态数据包括(翻滚角、俯仰角、倾斜角)。第一相机位姿数据包括第一相机光心坐标和第一相机姿态数据，第一相机姿态数据用于表示第一相机姿态。
步骤202、在第一图像帧中确定数字内容的第一跟踪点坐标。
第一跟踪点坐标对应于数字内容在场景中的位置，该位置也称为背景点。数字内容包括以数字形式存在的文本、图象、声音、三维物体等。
对于具有触控屏的计算装置，计算装置可以根据作用于屏幕的触控操作确定触控点坐标；将触控点坐标转换为在第一图像帧中数字内容的第一跟踪点坐标。或者，计算装置根据作用于屏幕的触控操作确定第一图像帧中数字内容的第一跟踪点坐标为预设坐标。触控操作可以是但不限于点击操作或者拖曳操作。
需要说的是，在生成第一跟踪点坐标之后，还可以生成包括第一跟踪点坐标的图像跟踪区域。可选的，图像跟踪区域可以是以第一跟踪点坐标为中心的区域。
本申请中，不同图像帧的图像跟踪区域对应同一个背景区域。具体的，图像跟踪区域可以是矩形，正方形或圆形，也可以其他形状。如图3所示，在一个示例中，用户在屏幕上点击后，根据点击位置生成第一跟踪点坐标31和图像跟踪区域32。
步骤203、生成包括第一相机光心的第一封闭空间。
具体可以根据封闭空间参数生成封闭空间。封闭空间参数值可以是预设的，也可以是由用户输入。封闭空间参数包括形状类型和空间尺寸参数，形状类型可以是立方体，正方体，圆柱体或圆锥体等。每种形状可以配置相应的空间尺寸参数，例如立方体的空间尺寸参数包括长度，宽度和高度。正方体的空间尺寸参数包括边长。圆柱体的空间参数包括底面半径和高度。封闭空间形状和空间尺寸参数可以根据实际情况进行选择，本申请不作限定。
步骤204、确定第一虚拟锚点为第一封闭空间的表面与第一射线的交点。
第一射线是在第一相机姿态下从第一相机光心坐标发出且经过第一跟踪点坐标的射线。图4为封闭空间和锚点的一个示意图。参阅图4，在一个示例中，封闭空间41为立方体，从相机光心401发出的射线经过跟踪点坐标402之后，与封闭空间41的表面交汇于虚拟锚点403，在背景42中的背景点421位于该射线的延长线上。
步骤205、根据第一相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第一图像帧上，以得到增强现实图像帧。应理解，投影后可以执行渲染等步骤，本申请不作限定。步骤202和步骤203并无固定先后顺序。
本实施例中，能够根据实际需要生成封闭空间，从第一相机光心坐标发出的射线与第一封闭空间的表面之间的交点作为数字内容的第一虚拟锚点，根据第一相机位姿数据和第一虚拟锚点能够将数字内容投影到第一图像帧，从而得到增强现实图像。这样不依赖背景点云生成锚点，在没有环境数据时也可以在任意位置摆放数字内容，而且能够将数字内容与背景贴合，因此具有更好的适用性。
本申请还可以对第一图像帧之后的其他图像帧进行增强现实处理。参阅图5，在另一个实施例中，本申请的增强现实方法还包括：
步骤501、在第一图像帧中确定包括第一跟踪点坐标的图像跟踪区域。
步骤502、使用相机摄取第二图像帧以及获取第二图像帧对应的第二相机位姿数据。第二图像帧可以是第一图像帧之后的任意一个图像帧。第二相机位姿数据包括第二相机光心坐标和第二相机姿态数据，第二相机姿态数据对应于第二相机姿态。
步骤503、根据图像跟踪区域确定在第二图像帧中的第二跟踪点坐标。
可选的，根据第一图像帧的图像跟踪区域可以确定第二图像帧的图像跟踪区域，然后确定第二跟踪点坐标为第二图像帧的图像跟踪区域的中心坐标。
步骤504、根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标是否能够确定背景点坐标，若是，执行步骤505，若否，执行步骤507。
步骤505、根据第一相机光心坐标，第一虚拟锚点的坐标，背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据。
具体的，第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标后，使用计算机视觉定位技术中的三角测量方法对上述数据进行处理，可以得到背景点坐标。
可选的，步骤505包括：根据第一相机光心坐标和第一虚拟锚点的坐标确定第一距离；根据第一相机光心坐标和背景点坐标确定第二距离；确定尺度对齐比例为第一距离与第二距离之比；根据第一相机光心坐标，尺度对齐比例和第二相机光心坐标计算第一虚拟相机光心坐标；根据第一虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第一虚拟相机位姿数据。应理解，第二图像帧对应的第一虚拟相机位姿数据包括第一虚拟相机光心坐标和第二相机姿态数据。
具体的，将第一距离记为Dv，第二距离记为Dt，尺度对齐比例记为Scale，它们满足以下公式：Scale＝Dv/Dt。
将尺度对齐比例记为Scale，第一相机光心坐标记为refPosition，第二相机光心坐标记为currPosition，虚拟相机光心的坐标记为virtualPosition，它们满足以下公式：
virtualPosition＝Scale*(currPosition-refPosition)。
参阅图6，在一个示例中，封闭空间61为立方体，相机光心601到虚拟锚点603的距离为Dv，相机光心601到背景点621的距离为Dt，可得Scale＝Dv/Dt。
相机光心601的坐标为refPosition，相机光心631的坐标为currPosition，按照virtualPosition＝Scale*(currPosition-refPosition)可以计算出虚拟相机光心632的坐标为virtualPosition。虚拟相机光心632对应的虚拟相机姿态数据与相机光心631对应的相机姿态数据相同。
由于第一虚拟相机光心到第一虚拟锚点的距离与第二相机光心到背景点的距离之间的比例为尺度对齐比例，因此在投影时数字内容的缩放比例与背景的缩放比例相同，从而使得数字内容与背景贴合。
步骤506、根据第二图像帧对应的第一虚拟相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第二图像帧上。
步骤507、根据预设的尺度对齐比例、第一相机光心坐标和第二相机光心坐标确定第二虚拟相机光心坐标。预设的尺度对齐比例Sc可以是0。按照virtualPosition＝Scale*(currPosition-refPosition)可以计算出第二虚拟相机光心坐标。
需要说明的是，本申请也可以不执行步骤507，直接将第二虚拟相机光心坐标设置为预设的虚拟相机光心坐标，预设的虚拟相机光心坐标可以根据实际情况进行设置，本申请不作限定。
步骤508、根据第二虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第二虚拟相机位姿数据。第二虚拟相机位姿数据包括第二虚拟相机光心坐标和第二相机姿态数据。
步骤509、根据第二图像帧对应的第二虚拟相机位姿数据和第二跟踪点坐标确定第三射线。第三射线是在第二相机姿态下从第二虚拟相机光心坐标发出且经过第二跟踪点坐标的射线。
步骤510、确定第三虚拟锚点为第一封闭空间的表面与第三射线的交点。
步骤511、根据第二图像帧对应的第二虚拟相机位姿和第三虚拟锚点的坐标将数字内容投影在第二图像帧上。
下面以一个示意图对第二虚拟相机位姿和第三虚拟锚点进行介绍，参阅图7，在一个示例中，封闭空间71为立方体，相机光心701，虚拟锚点703和背景点721在一条直线上。当相机处于相机光心731时，可以获取在相机光心731处的相机位姿数据，然后根据相机光心731对应的相机姿态数据和虚拟相机光心741的坐标可以确定虚拟相机位姿，在该虚拟相机位姿下根据虚拟相机光心741和跟踪点742确定射线743，将射线743与封闭空间71的交点作为虚拟锚点744。然后根据虚拟相机位姿和虚拟锚点744的坐标将数字内容投影，从而得到增强现实图像帧。
需要说明的是，步骤501可以在步骤202之后执行。步骤502在步骤205之后执行。图像跟踪区域可参阅图2所示实施例中的相应记载。
本实施例中，摄取第二图像帧之后，可以确定第二图像帧对应的第一虚拟相机位姿以及尺度对齐比例，根据第一虚拟锚点，第二图像帧对应的第一虚拟相机位姿以及尺度对齐比例能够实现增强现实功能，而且能够按照尺度对齐比例缩放数字内容，使得数字内容与背景贴合。
其次，在没有成功计算出背景点坐标的情况下也能获取第二虚拟相机位姿和第三虚拟锚点的坐标，从而实现增强现实功能，因此提高了方案实施的灵活性。当第三虚拟锚点与第一虚拟锚点处于第一封闭空间的同一平面时，可以实现数字内容与背景的缩放比例一致，使数字内容与背景贴合。
在实际应用中，相机位姿计算可能有误差，跟踪点坐标也可能出现误差，这些误差可能导致尺度对齐比例和虚拟相机位姿不正确，导致数字内容与背景没有贴合。本申请对此可以进行修正，下面对修正过程进行介绍，在一个可选实施例中，本申请的增强现实方法还包括：当数字内容与第二图像帧的背景区域没有贴合时，根据第二跟踪点坐标和第二图像帧对应的第一虚拟相机位姿数据确定第二射线；确定第二虚拟锚点为第一封闭空间的表面与第二射线的交点；根据第二图像帧对应的第一虚拟相机位姿数据和第二虚拟锚点的坐标将数字内容投影在第二图像帧上。
其中，第二射线是在第二相机姿态下从第一虚拟相机光心坐标发出且经过第二跟踪点坐标的射线。
本实施例中，当数字内容与第二图像帧的背景区域没有贴合时，可以采用第二图像帧对应的第一虚拟相机姿态数据和第二跟踪点坐标重新确定虚拟锚点，以得到目标增强现实图像帧。本申请可以不显示步骤506生成的增强现实图像帧且显示目标增强现实图像帧，能够减少数据内容与背景不贴合的图像帧。
参阅图8和图9，在一个示例中，相机中心801、虚拟锚点803和虚拟相机光心832的位置如图8所示。根据虚拟锚点803和虚拟相机光心832投影时，投影点为图9所示的投影点912。投影点912不是图像跟踪区域92的中心点，因此卡通松鼠不能与背景贴合。
当检测到卡通松鼠与背景不贴合时，将通过跟踪点841和虚拟相机光心832的射线与封闭空间81 的交点作为虚拟锚点842，根据虚拟锚点842和虚拟相机光心832将虚拟松鼠进行投影，投影点为图9所示的投影点911，投影点911是图像跟踪区域92的中心点，由此能够纠正投影位置的偏差。
当用户要调整数字内容的位置时，用户可以重新选择数字内容的摆放位置，而且可以选择封闭空间参数生成新的封闭空间。下面对其进行介绍，在一个可选实施例中，本申请的增强现实方法还包括：使用相机摄取第三图像帧以及获取第三图像帧对应的第三相机位姿数据，在第三图像帧中确定数字内容的第三跟踪点坐标；确定第四射线是在第三相机姿态下从第三相机光心坐标发出且经过第三跟踪点坐标的射线，生成包括第三相机光心的第二封闭空间；确定第四虚拟锚点为第二封闭空间与第四射线的交点。其中，第三相机位姿数据包括第三相机光心坐标和第三相机姿态数据，第三相机位姿数据对应于第三相机姿态。依此可以更新封闭空间、虚拟锚点以及虚拟相机光心，然后根据更新后的虚拟锚点和更新后的虚拟相机光心实现增强现实功能。本实施例中生成封闭空间，计算虚拟锚点的具体过程可参阅图2所示实施例的相应记载，此处不再赘述。
本申请还提供一种计算装置1000能够实现上述实施例中的增强现实方法。参阅图10，在一个实施例中，计算装置1000包括相机1001，位姿传感器1002和处理器1003。
相机1001用于摄取第一图像帧；位姿传感器1002用于获取第一图像帧对应的第一相机位姿数据，处理器1003用于在第一图像帧中确定数字内容的第一跟踪点坐标；生成包括第一相机光心的第一封闭空间；将在第一相机姿态下从第一相机光心坐标发出且经过第一跟踪点坐标的射线作为第一射线，确定第一虚拟锚点为第一封闭空间的表面与第一射线的交点，根据第一相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第一图像帧上，以得到增强现实图像帧。
在一个可选实施例中，处理器1003还用于在第一图像帧中确定包括第一跟踪点坐标的图像跟踪区域；相机1001还用于摄取第二图像帧；位姿传感器1002还用于获取第二图像帧对应的第二相机位姿数据；处理器1003还用于根据图像跟踪区域确定在第二图像帧中的第二跟踪点坐标；当根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标确定背景点坐标时，根据第一相机光心坐标，第一虚拟锚点的坐标，背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据；根据第二图像帧对应的第一虚拟相机位姿数据和第一虚拟锚点的坐标将数字内容投影在第二图像帧上。
在另一个可选实施例中，处理器1003具体用于根据第一相机光心坐标和第一虚拟锚点的坐标确定第一距离；根据第一相机光心坐标和背景点坐标确定第二距离；确定尺度对齐比例为第一距离与第二距离之比；根据第一相机光心坐标，尺度对齐比例和第二相机光心坐标计算第一虚拟相机光心坐标；根据第一虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第一虚拟相机位姿数据。
在另一个可选实施例中，处理器1003还用于当数字内容与第二图像帧的背景区域没有贴合时，根据第二跟踪点坐标和第二图像帧对应的第一虚拟相机位姿数据确定第二射线；确定第二虚拟锚点为第一封闭空间的表面与第二射线的交点；根据第二图像帧对应的第一虚拟相机位姿数据和第二虚拟锚点的坐标将数字内容投影在第二图像帧上。
在另一个可选实施例中，处理器1003还用于当根据第一相机位姿数据，第一跟踪点坐标，第二相机位姿数据和第二跟踪点坐标不能确定背景点坐标时，根据预设的尺度对齐比例、第一相机光心坐标和第二相机光心坐标确定第二虚拟相机光心坐标；根据第二虚拟相机光心坐标和第二相机姿态数据确定第二图像帧对应的第二虚拟相机位姿数据；根据第二图像帧对应的第二虚拟相机位姿数据和第二跟踪点坐标确定第三射线；确定第三虚拟锚点为第一封闭空间的表面与第三射线的交点；根据第二图像帧对应的第二虚拟相机位姿和第三虚拟锚点的坐标将数字内容投影在第二图像帧上。
在一个可选实施例中，处理器1003具体用于根据作用于屏幕的触控操作确定触控点坐标；将触控点坐标转换为在第一图像帧中数字内容的第一跟踪点坐标。
在另一个可选实施例中，处理器1003具体用于根据作用于屏幕的触控操作确定第一图像帧中数字内容的第一跟踪点坐标为预设坐标。
在另一个可选实施例中，相机1001还用于摄取第三图像帧；位姿传感器1002还用于获取第三图像帧对应的第三相机位姿数据，第三相机位姿数据包括第三相机光心坐标和第三相机姿态数据；处理器 1003还用于在第三图像帧中确定数字内容的第三跟踪点坐标；生成包括第三相机光心的第二封闭空间；确定第四射线是在第三相机姿态下从第三相机光心坐标发出且经过第三跟踪点坐标的射线，确定第四虚拟锚点为第二封闭空间与第四射线的交点。
本实施例中，位姿传感器1002执行的步骤可以由相机1001实现。本实施例中的名词解释，相机1001，位姿传感器1002和处理器1003执行的步骤和有益效果可参阅图2所示实施例、图5所示实施例或可选实施例中的相应记载。
本申请还提供一种计算装置1100能够实现上述实施例中的增强实现方法。如图11所示，计算装置1100包括：相机1101，深度传感器1102，处理器1103，惯性测量单元1104，输入单元1105，显示单元1106，存储器1107，通信接口1108和磁力计1109。相机1101，深度传感器1102，惯性测量单元1104，输入单元1105，显示单元1106，存储器1107，通信接口1108和磁力计1109与处理器1103之间通过总线连接。
总线可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。总线可包括在计算装置1100的各部件之间传送信息的通路。
处理器1103可以包括中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
深度传感器1102用于测量场景的深度信息。深度传感器1102可以是但不限于激光传感器。
惯性测量单元1104用于获取计算装置的运动数据，如线加速度，角速度等。
输入单元1105可以是触控屏，键盘，鼠标或其他输入设备。
显示单元1106可以是显示器。
磁力计1109用于测量计算装置的偏航角。
存储器1107可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。处理器104还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)。
存储器1107中存储有可执行的程序代码，处理器1103执行该可执行的程序代码以分别实现前述相机、位姿传感器和处理器的功能，从而实现增强现实方法。即存储器1107存储有用于执行增强现实方法的指令。
通信接口1108使用例如但不限于网络接口卡、收发器一类的收发模块，来实现计算装置1100与其他设备或通信网络之间的通信。
应理解，本申请不限定计算装置1100中的相机，深度传感器，处理器，惯性测量单元，输入单元，显示单元，存储器，通信接口和磁力计的个数。
本申请实施例还提供了一种包含指令的计算机程序产品。所述计算机程序产品可以是包含指令的，能够运行在计算装置上或被储存在任何可用介质中的软件或程序产品。当所述计算机程序产品在至少一个计算装置上运行时，使得至少一个计算装置执行增强现实方法。
本申请实施例还提供了一种计算机可读存储介质。所述计算机可读存储介质可以是计算装置能够存储的任何可用介质或者是包含一个或多个可用介质的数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘)等。该计算机可读存储介质包括指令，所述指令指示计算装置执行增强现实方法。
在本申请的各个实施例中，如果没有特殊说明以及逻辑冲突，不同的实施例之间的术语和/或描述具有一致性、且可以相互引用，不同的实施例中的技术特征根据其内在的逻辑关系可以组合形成新的实施例。
可以理解的是，在本申请的实施例中涉及的各种数字编号仅为描述方便进行的区分，并不用来限制本申请的实施例的范围。上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以 其功能和内在逻辑确定。
以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例记载的技术方案的范围。

Claims (19)

  1. 一种增强现实方法，其特征在于，包括：
    使用相机摄取第一图像帧以及获取所述第一图像帧对应的第一相机位姿数据，所述第一相机位姿数据包括第一相机光心坐标和第一相机姿态数据；
    在所述第一图像帧中确定数字内容的第一跟踪点坐标；
    生成包括第一相机光心的第一封闭空间；
    确定第一虚拟锚点为所述第一封闭空间的表面与第一射线的交点，所述第一射线是在第一相机姿态下从所述第一相机光心坐标发出且经过所述第一跟踪点坐标的射线；
    根据所述第一相机位姿数据和所述第一虚拟锚点的坐标将所述数字内容投影在所述第一图像帧上，以得到增强现实图像帧。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    在所述第一图像帧中确定包括所述第一跟踪点坐标的图像跟踪区域；
    使用相机摄取第二图像帧以及获取所述第二图像帧对应的第二相机位姿数据，所述第二相机位姿数据包括第二相机光心坐标和第二相机姿态数据；
    根据所述图像跟踪区域确定在所述第二图像帧中的第二跟踪点坐标；
    当根据所述第一相机位姿数据，所述第一跟踪点坐标，所述第二相机位姿数据和所述第二跟踪点坐标确定背景点坐标时，根据第一相机光心坐标，所述第一虚拟锚点的坐标，所述背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据；
    根据所述第二图像帧对应的第一虚拟相机位姿数据和所述第一虚拟锚点的坐标将所述数字内容投影在所述第二图像帧上。
  3. 根据权利要求2所述的方法，其特征在于，所述根据第一相机光心坐标，所述第一虚拟锚点的坐标，所述背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据包括：
    根据所述第一相机光心坐标和所述第一虚拟锚点的坐标确定第一距离；
    根据所述第一相机光心坐标和所述背景点坐标确定第二距离；
    确定尺度对齐比例为所述第一距离与所述第二距离之比；
    根据所述第一相机光心坐标，所述尺度对齐比例和所述第二相机光心坐标计算第一虚拟相机光心坐标；
    根据所述第一虚拟相机光心坐标和所述第二相机姿态数据确定第二图像帧对应的第一虚拟相机位姿数据。
  4. 根据权利要求3所述的方法，其特征在于，所述方法还包括：
    当所述数字内容与所述第二图像帧的背景区域没有贴合时，根据所述第二跟踪点坐标和所述第二图像帧对应的第一虚拟相机位姿数据确定第二射线；
    确定第二虚拟锚点为所述第一封闭空间的表面与第二射线的交点；
    根据第二图像帧对应的第一虚拟相机位姿数据和所述第二虚拟锚点的坐标将所述数字内容投影在所述第二图像帧上。
  5. 根据权利要求2所述的方法，其特征在于，所述方法还包括：
    当根据所述第一相机位姿数据，所述第一跟踪点坐标，所述第二相机位姿数据和所述第二跟踪点坐标不能确定背景点坐标时，根据预设的尺度对齐比例、第一相机光心坐标和所述第二相机光心坐标确定第二虚拟相机光心坐标；
    根据所述第二虚拟相机光心坐标和所述第二相机姿态数据确定第二图像帧对应的第二虚拟相机位姿数据；
    根据所述第二图像帧对应的第二虚拟相机位姿数据和所述第二跟踪点坐标确定第三射线；
    确定第三虚拟锚点为所述第一封闭空间的表面与所述第三射线的交点；
    根据所述第二图像帧对应的第二虚拟相机位姿和所述第三虚拟锚点的坐标将所述数字内容投影在所述第二图像帧上。
  6. 根据权利要求1至5中任一项所述的方法，其特征在于，在所述第一图像帧中确定数字内容的第一跟踪点坐标包括：
    根据作用于屏幕的触控操作确定触控点坐标；
    将所述触控点坐标转换为在所述第一图像帧中数字内容的第一跟踪点坐标。
  7. 根据权利要求1至5中任一项所述的方法，其特征在于，在所述第一图像帧中确定数字内容的第一跟踪点坐标包括：
    根据作用于屏幕的触控操作确定所述第一图像帧中数字内容的第一跟踪点坐标为预设坐标。
  8. 根据权利要求1至5中任一项所述的方法，其特征在于，所述方法还包括：
    使用相机摄取第三图像帧以及获取所述第三图像帧对应的第三相机位姿数据，所述第三相机位姿数据包括第三相机光心坐标和第三相机姿态数据；
    在所述第三图像帧中确定数字内容的第三跟踪点坐标；
    生成包括第三相机光心的第二封闭空间；
    确定第四虚拟锚点为所述第二封闭空间与第四射线的交点，所述第四射线是在第三相机姿态下从所述第三相机光心坐标发出且经过第三跟踪点坐标的射线。
  9. 一种计算装置，其特征在于，包括：
    相机，用于摄取第一图像帧；
    位姿传感器，用于获取所述第一图像帧对应的第一相机位姿数据，所述第一相机位姿数据包括第一相机光心坐标和第一相机姿态数据；
    处理器，用于在所述第一图像帧中确定数字内容的第一跟踪点坐标；生成包括第一相机光心的第一封闭空间；确定第一虚拟锚点为所述第一封闭空间的表面与第一射线的交点，所述第一射线是在第一相机姿态下从所述第一相机光心坐标发出且经过所述第一跟踪点坐标的射线；根据所述第一相机位姿数据和所述第一虚拟锚点的坐标将所述数字内容投影在所述第一图像帧上，以得到增强现实图像帧。
  10. 根据权利要求9所述的计算装置，其特征在于，
    所述处理器还用于在所述第一图像帧中确定包括所述第一跟踪点坐标的图像跟踪区域；
    所述相机还用于摄取第二图像帧；
    所述位姿传感器还用于获取所述第二图像帧对应的第二相机位姿数据，所述第二相机位姿数据包括第二相机光心坐标和第二相机姿态数据；
    所述处理器还用于根据所述图像跟踪区域确定在所述第二图像帧中的第二跟踪点坐标；当根据所述第一相机位姿数据，所述第一跟踪点坐标，所述第二相机位姿数据和所述第二跟踪点坐标确定背景点坐标时，根据第一相机光心坐标，所述第一虚拟锚点的坐标，所述背景点坐标和第二相机位姿数据确定第二图像帧对应的第一虚拟相机位姿数据；根据所述第二图像帧对应的第一虚拟相机位姿数据和所述第一虚拟锚点的坐标将所述数字内容投影在所述第二图像帧上。
  11. 根据权利要求10所述的计算装置，其特征在于，所述处理器具体用于根据所述第一相机光心坐标和所述第一虚拟锚点的坐标确定第一距离；根据所述第一相机光心坐标和所述背景点坐标确定第二距离；确定尺度对齐比例为所述第一距离与所述第二距离之比；根据所述第一相机光心坐标，所述尺度对齐比例和所述第二相机光心坐标计算第一虚拟相机光心坐标；根据所述第一虚拟相机光心坐标和所述第二相机姿态数据确定第二图像帧对应的第一虚拟相机位姿数据。
  12. 根据权利要求11所述的计算装置，其特征在于，所述处理器还用于当所述数字内容与所述第二图像帧的背景区域没有贴合时，根据所述第二跟踪点坐标和所述第二图像帧对应的第一虚拟相机位姿数据确定第二射线；确定第二虚拟锚点为所述第一封闭空间的表面与第二射线的交点；根据第二图像帧对应的第一虚拟相机位姿数据和所述第二虚拟锚点的坐标将所述数字内容投影在所述第二图像帧上。
  13. 根据权利要求10所述的计算装置，其特征在于，所述处理器还用于当根据所述第一相机位姿数据，所述第一跟踪点坐标，所述第二相机位姿数据和所述第二跟踪点坐标不能确定背景点坐标时，根据预设的尺度对齐比例、第一相机光心坐标和所述第二相机光心坐标确定第二虚拟相机光心坐标；根据所述第二虚拟相机光心坐标和所述第二相机姿态数据确定第二图像帧对应的第二虚拟相机位姿数据；根据 所述第二图像帧对应的第二虚拟相机位姿数据和所述第二跟踪点坐标确定第三射线；确定第三虚拟锚点为所述第一封闭空间的表面与所述第三射线的交点；根据所述第二图像帧对应的第二虚拟相机位姿和所述第三虚拟锚点的坐标将所述数字内容投影在所述第二图像帧上。
  14. 根据权利要求9至13中任一项所述的计算装置，其特征在于，所述处理器具体用于根据作用于屏幕的触控操作确定触控点坐标；将所述触控点坐标转换为在所述第一图像帧中数字内容的第一跟踪点坐标。
  15. 根据权利要求9至13中任一项所述的计算装置，其特征在于，所述处理器具体用于根据作用于屏幕的触控操作确定所述第一图像帧中数字内容的第一跟踪点坐标为预设坐标。
  16. 根据权利要求9至13中任一项所述的计算装置，其特征在于，
    所述相机还用于摄取第三图像帧；
    所述位姿传感器还用于获取所述第三图像帧对应的第三相机位姿数据，所述第三相机位姿数据包括第三相机光心坐标和第三相机姿态数据；
    所述处理器还用于在所述第三图像帧中确定数字内容的第三跟踪点坐标；生成包括第三相机光心的第二封闭空间；确定第四虚拟锚点为所述第二封闭空间与第四射线的交点，所述第四射线是在第三相机姿态下从所述第三相机光心坐标发出且经过第三跟踪点坐标的射线。
  17. 一种计算装置，其特征在于，包括处理器和存储器，存储器用于存储指令，处理器用于执行所述指令，以使得所述计算装置执行权利要求1至8中任一项所述的方法。
  18. 一种计算机可读存储介质，存储有指令，其特征在于，当所述指令由计算机运行时，所述计算机执行权利要求1至8中任一项所述的方法。
  19. 一种包含指令的计算机程序产品，其特征在于，当所述指令被计算机运行时，使得所述计算机执行权利要求1至8中任一项所述的方法。
PCT/CN2023/103178 2022-09-16 2023-06-28 一种增强现实方法和计算装置 WO2024055683A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211130875.4A CN117768627A (zh) 2022-09-16 2022-09-16 一种增强现实方法和计算装置
CN202211130875.4 2022-09-16

Publications (1)

Publication Number Publication Date
WO2024055683A1 true WO2024055683A1 (zh) 2024-03-21

Family

ID=90274156

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/103178 WO2024055683A1 (zh) 2022-09-16 2023-06-28 一种增强现实方法和计算装置

Country Status (2)

Country Link
CN (1) CN117768627A (zh)
WO (1) WO2024055683A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110286768A (zh) * 2019-06-27 2019-09-27 Oppo广东移动通信有限公司 虚拟物体显示方法、终端设备及计算机可读存储介质
CN110926334A (zh) * 2019-11-29 2020-03-27 深圳市商汤科技有限公司 测量方法、装置、电子设备及存储介质
US10832417B1 (en) * 2019-06-04 2020-11-10 International Business Machines Corporation Fusion of visual-inertial-odometry and object tracker for physically anchored augmented reality
CN113741698A (zh) * 2021-09-09 2021-12-03 亮风台（上海）信息科技有限公司 一种确定和呈现目标标记信息的方法与设备
WO2022057308A1 (zh) * 2020-09-16 2022-03-24 北京市商汤科技开发有限公司 显示方法、装置，显示设备及计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10832417B1 (en) * 2019-06-04 2020-11-10 International Business Machines Corporation Fusion of visual-inertial-odometry and object tracker for physically anchored augmented reality
CN110286768A (zh) * 2019-06-27 2019-09-27 Oppo广东移动通信有限公司 虚拟物体显示方法、终端设备及计算机可读存储介质
CN110926334A (zh) * 2019-11-29 2020-03-27 深圳市商汤科技有限公司 测量方法、装置、电子设备及存储介质
WO2022057308A1 (zh) * 2020-09-16 2022-03-24 北京市商汤科技开发有限公司 显示方法、装置，显示设备及计算机可读存储介质
CN113741698A (zh) * 2021-09-09 2021-12-03 亮风台（上海）信息科技有限公司 一种确定和呈现目标标记信息的方法与设备

Also Published As

Publication number Publication date
CN117768627A (zh) 2024-03-26

Similar Documents

Publication Publication Date Title
US9516214B2 (en) Information processing device and information processing method
US11625841B2 (en) Localization and tracking method and platform, head-mounted display system, and computer-readable storage medium
CN109668545B (zh) 用于头戴式显示装置的定位方法、定位器以及定位系统
JP6500355B2 (ja) 表示装置、表示プログラム、および表示方法
WO2019118877A1 (en) Spherical video editing
TWI677413B (zh) 用於機械手臂系統之校正方法及裝置
US10825249B2 (en) Method and device for blurring a virtual object in a video
US20170374331A1 (en) Auto keystone correction and auto focus adjustment
WO2021031781A1 (zh) 投影图像校准方法、装置及投影设备
WO2020024185A1 (en) Techniques for motion-based automatic image capture
WO2022022141A1 (zh) 图像显示方法、装置、计算机设备及存储介质
WO2022022449A1 (zh) 用于空间定位的方法和装置
CN108961423B (zh) 虚拟信息处理方法、装置、设备及存储介质
US20160062486A1 (en) Mobile device and method of projecting image by using the mobile device
US20200349727A1 (en) Obstacle detection method, mobile platform, and computer readable storage medium
JP7195238B2 (ja) 拡張現実アプリケーションに関するシステム及び方法
JP2020113094A (ja) 拡張現実空間に配置される３ｄオブジェクトを生成する方法
KR20010055957A (ko) 증강현실 기반의 3차원 트래커와 컴퓨터 비젼을 이용한영상 정합 방법
WO2024055683A1 (zh) 一种增强现实方法和计算装置
CN113034347A (zh) 倾斜摄影图像处理方法、装置、处理设备及存储介质
WO2023160301A1 (zh) 物体信息确定方法、移动机器人系统及电子设备
US10735665B2 (en) Method and system for head mounted display infrared emitter brightness optimization based on image saturation
US20220191542A1 (en) Object Pose Estimation and Tracking Using Machine Learning
CN115393469A (zh) 房屋户型图生成方法、装置、设备及介质
KR102398404B1 (ko) 객체 제어를 지원하기 위한 방법, 시스템 및 비일시성의 컴퓨터 판독 가능 기록 매체

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23864422

Country of ref document: EP

Kind code of ref document: A1