WO2021082801A1 - Procédé et appareil de traitement de réalité augmentée, système, support d'enregistrement et dispositif électronique - Google Patents

Procédé et appareil de traitement de réalité augmentée, système, support d'enregistrement et dispositif électronique Download PDF

Info

Publication number
WO2021082801A1
WO2021082801A1 PCT/CN2020/116290 CN2020116290W WO2021082801A1 WO 2021082801 A1 WO2021082801 A1 WO 2021082801A1 CN 2020116290 W CN2020116290 W CN 2020116290W WO 2021082801 A1 WO2021082801 A1 WO 2021082801A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature point
current frame
point information
frame image
dimensional feature
Prior art date
Application number
PCT/CN2020/116290
Other languages
English (en)
Chinese (zh)
Inventor
黄锋华
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2021082801A1 publication Critical patent/WO2021082801A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation

Definitions

  • the present disclosure relates to the field of augmented reality technology, and in particular to an augmented reality processing method, an augmented reality processing device, an augmented reality processing system, a storage medium, and electronic equipment.
  • Augmented Reality is a technology that integrates the virtual world and the real world. This technology has been widely used in education, games, medical care, the Internet of Things, intelligent manufacturing and other fields.
  • the relocation effect plays a crucial role in the AR experience.
  • the shooting angles of the mapping device and the relocation device are not the same, in the process of determining the pose relationship between the mapping device and the relocation device, the problem of feature mismatch may occur, resulting in poor relocation results .
  • an augmented reality processing method which is applied to a first device, and the augmented reality processing method includes: determining image parameters of a current frame image of the first device; and acquiring the image parameters determined by the second device Image parameters of the reference image; determine the pose of the current frame image relative to the second device based on the image parameters of the current frame image and the image parameters of the reference image; capture the current frame image based on the pose of the current frame image relative to the second device
  • the posture information of the first device is used, the relative posture relationship between the first device and the second device is determined, so as to use the relative posture relationship between the first device and the second device to perform augmented reality processing operations.
  • an augmented reality processing device applied to a first device.
  • the augmented reality processing device includes a first image parameter determination module, a second image parameter determination module, and a first relative pose determination module And the second relative pose determination module.
  • the first image parameter determination module is used to determine the image parameters of the current frame image of the first device; the second image parameter determination module is used to obtain the image parameters of the reference image determined by the second device; the first relative pose The determining module is used to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image; the second relative pose determining module is used to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image relative to the second device.
  • the pose and the pose information of the first device when the current frame image is collected determine the relative pose relationship between the first device and the second device, so as to use the relative pose relationship between the first device and the second device to perform augmented reality processing operations.
  • an augmented reality processing system applied to a first device.
  • the augmented reality processing system includes a camera module, a depth sensing module, an inertial measurement unit, a real-time positioning and map construction unit, and Augmented reality processing device.
  • the camera module is used to collect the current frame image; the depth sensing module is used to collect the depth information corresponding to the current frame image; the inertial measurement unit is used to measure the inertial information of the first device; the instant positioning and map construction unit is used To obtain the current frame image and inertial information, and generate the posture information of the first device based on the inertial information; the augmented reality processing device is used to determine the image parameters of the current frame image; to obtain the image parameters of the reference image determined by the second device; The image parameters of the current frame image and the image parameters of the reference image determine the pose of the current frame image relative to the second device, and combine the posture information of the first device when the current frame image is collected to determine the relative position of the first device and the second device Posture relationship.
  • a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned augmented reality processing method is implemented.
  • an electronic device including a processor; and a memory for storing executable instructions of the processor; the processor is configured to execute the aforementioned augmented reality processing by executing the executable instructions method.
  • FIG. 1 shows a schematic diagram of a scenario architecture suitable for implementing exemplary embodiments of the present disclosure
  • Fig. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present invention
  • FIG. 3 schematically shows an architecture diagram of an augmented reality processing system according to an exemplary embodiment of the present disclosure
  • FIG. 4 schematically shows a flowchart of an augmented reality processing method according to an exemplary embodiment of the present disclosure
  • FIG. 5 schematically shows a flowchart of determining the relative pose relationship between the first device and the second device by using the iterative closest point method according to the present disclosure
  • Fig. 6 schematically shows a block diagram of an augmented reality processing device according to an exemplary embodiment of the present disclosure
  • FIG. 7 schematically shows a block diagram of a first relative pose determination module according to an exemplary embodiment of the present disclosure
  • Fig. 8 schematically shows a block diagram of a first relative pose determination module according to another exemplary embodiment of the present disclosure
  • FIG. 9 schematically shows a block diagram of an augmented reality processing apparatus according to another exemplary embodiment of the present disclosure.
  • Fig. 1 shows a schematic diagram of a scenario architecture suitable for implementing exemplary embodiments of the present disclosure.
  • the augmented reality processing solution architecture of the exemplary embodiment of the present disclosure may include a first device 1001 and a second device 1002.
  • the second device 1002 is used to map the scene in which it is located
  • the first device 1001 is a terminal device that is currently to perform an augmented reality processing operation in the scene.
  • the first device 1001 and the second device 1002 may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, smart wearable devices, and the like.
  • the first device 1001 and the second device 1002 may be communicatively connected. Specifically, the connection can be established through Bluetooth, hotspot, WiFi, mobile network, etc., so that the second device 1002 can directly transmit data with the first device 1001 without the data going through the server.
  • the video frame image can be obtained based on the camera module of the second device 1002, and the depth information corresponding to each video frame image can be obtained based on the depth sensing module.
  • the second device 1002 can determine two-dimensional feature point information and three-dimensional feature point information.
  • the second device 1002 can send the two-dimensional feature point information and three-dimensional feature point information of each frame of image or key frame image to the first One device 1001.
  • the first device 1001 When the first device 1001 performs the augmented reality processing operation, the first device 1001 can obtain the current frame image taken by its own camera module, and based on the corresponding depth information, determine the two-dimensional feature point information and the three-dimensional feature point information of the current frame image. Feature point information. Subsequently, the first device 1001 may match the two-dimensional feature point information and the three-dimensional feature point information of the current frame image with the two-dimensional feature point information and the three-dimensional feature point information of the image determined by the second device 1002, and determine the current frame based on the matching result The pose of the image relative to the second device 1002. Next, the first device 1001 combines its own posture information to determine the relative posture relationship between the first device 1001 and the second device 1002.
  • the first device 1001 can obtain the anchor point information, so that the first device 1001 and the second device 1002 can be in the same position in the scene Display virtual objects and perform other augmented reality processing operations.
  • the second device 1002 can also send information to the first device 1001 by means of the server 1003.
  • the server 1003 may be a cloud server.
  • the second device 1002 may send the two-dimensional feature point information and the three-dimensional feature point information of each frame of image in the mapping process to the server 1003, and may also send the configured anchor point information to the server 1003.
  • the first device 1001 can obtain the two-dimensional feature point information and the three-dimensional feature point information of each frame of image during the mapping process from the server 1003, and compare it with the current The information of the frame image is matched to determine the relative pose relationship between the first device 1001 and the second device 1002.
  • the number of terminal devices and servers in FIG. 1 is only illustrative, and any number of terminal devices and servers may be provided according to implementation needs.
  • the server 1003 may be a server cluster composed of multiple servers.
  • multi-person AR interaction with more than three persons can also be realized.
  • the terminal device used for mapping is described as the second device, and the terminal device currently used for processing operations of augmented reality is described as the first device for distinction. It should be understood that the second device may be a terminal device currently performing processing operations in some scenarios, and the first device may also be a terminal device for mapping in some scenarios.
  • the augmented reality processing method of the exemplary embodiment of the present disclosure is executed by the first device, and accordingly, the augmented reality processing device and the augmented reality processing system described below may be configured in the first device.
  • FIG. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an exemplary embodiment of the present disclosure. That is to say, FIG. 2 exemplarily shows a schematic diagram of the computer structure of the above-mentioned first device.
  • the computer system 200 includes a central processing unit (CPU) 201, which can be based on a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203 And perform various appropriate actions and processing.
  • ROM read-only memory
  • RAM random access memory
  • various programs and data required for system operation are also stored.
  • the CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204.
  • An input/output (I/O) interface 205 is also connected to the bus 204.
  • the following components are connected to the I/O interface 205: an input part 206 including a keyboard, a mouse, etc.; an output part 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and speakers, etc.; a storage part 208 including a hard disk, etc. ; And a communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the Internet.
  • the drive 210 is also connected to the I/O interface 205 as needed.
  • the removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that the computer program read from it is installed into the storage part 208 as needed.
  • an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart.
  • the computer program may be downloaded and installed from the network through the communication part 209, and/or installed from the removable medium 211.
  • CPU central processing unit
  • various functions defined in the system of the present application are executed.
  • the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
  • each block in the flowchart or block diagram may represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more for realizing the specified logic function.
  • Executable instructions may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by It is realized by a combination of dedicated hardware and computer instructions.
  • the units described in the embodiments of the present disclosure may be implemented in software or hardware, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.
  • this application also provides a computer-readable medium.
  • the computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist alone without being assembled into the electronic device. in.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device realizes the method described in the following embodiments.
  • the augmented reality processing system of the exemplary embodiment of the present disclosure may include an inertial measurement unit 31, a camera module 32, a depth sensing module group 33, an instant positioning and map construction unit 34, and an augmented reality processing device 35.
  • the inertial measurement unit 31 may include a gyroscope and an accelerometer, which respectively measure the angular velocity and acceleration of the first device, and thereby determine the inertial information of the first device.
  • the camera module 31 can be used to collect video frame images, where the video frame images are RGB images. During the execution of the following augmented reality processing, the camera module 31 can acquire the current frame of image for subsequent processing.
  • the depth sensing module 33 may be used to collect depth information.
  • the depth sensing module may be a dual camera module, a structured light module, or a TOF (Time-Of-Flight, Time-Of-Flight) module. This disclosure does not impose special restrictions on this.
  • the real-time positioning and map construction unit 34 can be used to obtain the inertial information sent by the inertial measurement unit 31 and the image sent by the camera module 32, and perform the mapping and relocation process.
  • the augmented reality processing device 35 can obtain the current frame image sent by the instant positioning and map construction unit 34, determine the image parameters of the current frame image, obtain the image parameters of the reference image determined by the second device, and determine the image parameters of the reference image according to the current frame image.
  • the image parameters and the image parameters of the reference image determine the pose of the current frame image relative to the second device, and combine the pose information of the first device when the current frame image is used to determine the relative pose relationship between the first device and the second device.
  • the augmented reality processing device 35 may obtain the current frame image and the corresponding posture information sent by the instant positioning and map construction unit 34. Extract the two-dimensional feature point information of the current frame image; obtain the depth information corresponding to the current frame image from the depth sensing module 33, and determine the three-dimensional feature point information of the current frame image according to the depth information corresponding to the two-dimensional feature point information; Use the two-dimensional feature point information and the three-dimensional feature point information of the current frame image and the two-dimensional feature point information and the three-dimensional feature point information of the image determined by the second device to determine the pose of the current frame image relative to the second device, and Combining the posture information of the first device, determine the relative posture relationship between the first device and the second device, so as to use the relative posture relationship between the first device and the second device to perform augmented reality processing operations.
  • the augmented reality processing device 35 may also include an anchor point acquisition module.
  • the anchor point acquisition module is used to acquire the anchor point information configured by the second device, so as to display the virtual object corresponding to the anchor point information on the first device based on the relative pose relationship between the first device and the second device.
  • the first device can also add anchor point information in the scene so that other devices can display and perform interactive operations.
  • the enhanced display processing system may further include an anchor point adding unit.
  • the anchor point adding unit may be used to add anchor point information in the scene where the first device is located.
  • the anchor point adding unit may include an application program 36 as shown in FIG. 3, and the user holding the first device may implement the addition of anchor point information by means of the application program 36.
  • the second device involved in the exemplary embodiment of the present disclosure may also have a system architecture as shown in FIG. 3.
  • the augmented reality processing method may include the following steps:
  • the image parameter of the current frame image may include two-dimensional feature point information and three-dimensional feature point information of the current frame image.
  • the first device can obtain the current frame image taken by the camera module, and perform feature extraction on the current frame image to determine the two-dimensional feature point information of the current frame image. Specifically, the two-dimensional feature point information of the current frame image can be extracted based on the combination of the feature extraction algorithm and the feature descriptor.
  • the feature extraction algorithms adopted by the exemplary embodiments of the present disclosure may include, but are not limited to, FAST feature point detection algorithm, DOG feature point detection algorithm, Harris feature point detection algorithm, SIFT feature point detection algorithm, SURF feature point detection algorithm, etc.
  • Feature descriptors may include, but are not limited to, BRIEF feature point descriptors, BRISK feature point descriptors, FREAK feature point descriptors, and so on.
  • the combination of the feature extraction algorithm and the feature descriptor may be the FAST feature point detection algorithm and the BRIEF feature point descriptor. According to other embodiments of the present disclosure, the combination of the feature extraction algorithm and the feature descriptor may be a DOG feature point detection algorithm and a FREAK feature point descriptor.
  • the first device may respond to the user's operation to perform the process of acquiring the current frame image and extracting the two-dimensional feature. For example, when the user starts the AR application, the first device may respond to the AR application startup operation, turn on the camera module, obtain the current frame image captured by the camera module, and extract the two-dimensional feature point information.
  • the depth information corresponding to the two-dimensional feature point information can be combined to determine the three-dimensional feature point information of the current frame image.
  • the depth information corresponding to the current frame image can be acquired through the depth sensing module.
  • the depth sensing module may be any one of a dual camera module (for example, a color camera and a telephoto camera), a structured light module, and a TOF module.
  • the current frame image and the corresponding depth information can be registered to determine the depth information of each pixel on the current frame image.
  • the internal and external parameters of the camera module and the depth sensing module need to be calibrated in advance.
  • the internal parameter matrix of the depth sensing module can be used to obtain the coordinate P_ir of the pixel point in the depth sensing module coordinate system.
  • P_ir can be multiplied by a rotation matrix R, and a translation vector T is added to convert P_ir to the coordinate system of the RGB camera to obtain P_rgb.
  • P_rgb can be multiplied with the internal parameter matrix H_rgb of the camera module to obtain p_rgb
  • p_rgb is also a three-dimensional vector, denoted as (x0, y0, z0), where x0 and y0 are the pixels of the pixel in the RGB image Coordinates, extract the pixel value of the pixel, and match it with the corresponding depth information.
  • the alignment of the two-dimensional image information and depth information of one pixel is completed. In this case, the above process is performed for each pixel to complete the registration process.
  • the depth information corresponding to the two-dimensional feature point information can be determined from it, and the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information can be combined to determine Get the three-dimensional feature point information of the current frame image.
  • the depth information can also be denoised to remove obviously wrong depth values in the depth information.
  • a deep neural network may be used to remove noise in the TOF image, which is not particularly limited in this exemplary embodiment.
  • the image parameters of the reference image may include two-dimensional feature point information and three-dimensional feature point information of the reference image.
  • the second device can generate two-dimensional feature point information and three-dimensional feature point information of each frame of image or key frame image.
  • the first device is in the scene created by the second device and performs an augmented reality processing operation
  • the two-dimensional feature point information and the three-dimensional feature point information of these images can be acquired.
  • the reference image described in the present disclosure is each frame image or key frame image generated by the second device in the mapping.
  • the second device can send the two-dimensional feature point information and three-dimensional feature point information of the reference image to the first device through Bluetooth, hotspot, WiFi, mobile network, etc. .
  • the second device may send the two-dimensional feature point information and the three-dimensional feature point information of the reference image to the cloud server, so that the first device can obtain the two-dimensional feature point information and the three-dimensional feature point information of the reference image from the cloud server.
  • Three-dimensional feature point information may be used to send the two-dimensional feature point information and the three-dimensional feature point information of the reference image to the cloud server.
  • step S42 and step S44 can be interchanged, that is, the solution of the present disclosure may also execute step S44 first, and then execute step S42.
  • the pose of the current frame image relative to the second device can be determined, that is, the pose of the current frame image in the second device coordinate system can be determined .
  • the present disclosure provides three implementation manners, which will be described one by one below.
  • the relationship between the two-dimensional feature point information of the current frame image and the two-dimensional feature point information of the reference image can be determined through feature matching or descriptor matching. If the two-dimensional feature point information of the current frame image is determined If the feature point information matches the two-dimensional feature point information of the reference image, the Iterative Closest Point (ICP) method can be used to determine the relative relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image. Posture relationship.
  • ICP Iterative Closest Point
  • the three-dimensional feature point information of the current frame image is the point cloud information corresponding to the current frame image
  • the three-dimensional feature point information of the reference image is the point cloud information of the reference image.
  • the two point cloud information can be used as input, and by inputting the specified pose as the initial value, the optimal relative pose after the alignment of the two point clouds is obtained by the iterative closest point method, that is, the three-dimensional feature of the current frame image is determined
  • the relative pose relationship between the point information and the three-dimensional feature point information of the reference image is determined.
  • the relationship between the two-dimensional information is determined first. Due to the determination of the two-dimensional information relationship, the method of feature matching or descriptor matching is usually adopted, and the process is simple. As a result, the entire matching process can be accelerated, and the accuracy can be improved, while the effect of troubleshooting in advance can also be achieved.
  • the exemplary embodiment of the present disclosure may also include a solution for removing mismatched points.
  • the RANSAC (Random Sample Consensus) method can be used to eliminate mismatched feature point information. Specifically, a certain number of matching pairs (for example, 7 pairs, 8 pairs, etc.) are randomly selected from the matching pairs between the two-dimensional feature points of the current frame image and the two-dimensional feature points of the reference image, and the selected matching pairs are calculated
  • the basic matrix or essential matrix between the current frame image and the reference image is based on the epipolar constraint. If a two-dimensional feature point is far from the corresponding epipolar line, for example, greater than a threshold, the two-dimensional feature can be considered The points are mismatched points. By iterating a certain number of random sampling processes, the random sampling result with the largest number of internal points is selected as the final matching result. On this basis, the mismatched feature point information can be eliminated from the three-dimensional feature point information of the current frame image.
  • the three-dimensional feature point information from which the mismatched feature point information is eliminated can be used to determine the pose of the current frame image relative to the second device.
  • the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image
  • the two-dimensional feature point information of the current frame image is matched with the three-dimensional feature point information of the reference image.
  • Information association to get point-to-point information
  • the point pair information can be used as input to solve the Perspective-n-Point (PnP) problem.
  • PnP Perspective-n-Point
  • PnP is a classic method in the field of machine vision, which can determine the relative pose between the camera and the object according to n feature points on the object. Specifically, the rotation matrix and translation vector between the camera and the object can be determined according to the n feature points on the object. In addition, n may be determined to be 4 or more, for example.
  • the relative pose relationship between the three-dimensional feature point information of the reference image and the three-dimensional feature point information obtained by combining the PnP solution result of the previous embodiment can be used as the iterative initial pose input.
  • the point method determines the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image to determine the pose of the current frame image relative to the second device. It is easy to see that in this embodiment, PnP is combined with ICP to improve the accuracy of determining the pose relationship.
  • the inertial measurement unit of the first device can obtain the inertial information of the first device, and thereby, the 6DOF (6 Degrees Of Freedom) attitude information of the first device can be obtained. Based on the posture information of the first device and the posture of the current frame image relative to the second device determined in step S46, the relative posture relationship between the first device and the second device can be obtained.
  • 6DOF Degrees Of Freedom
  • the first device may extract the two-dimensional feature point information of the current frame image.
  • the DOG feature point detection algorithm and the FREAK feature point descriptor may be used for feature extraction.
  • the first device can obtain the depth information input by the TOF module; in step S506, the two-dimensional feature point information can be registered with the depth information to obtain the point cloud data of the current frame image.
  • the first device may determine whether the two-dimensional feature point information determined in step S502 matches the two-dimensional feature of the reference image, and if it matches, it executes the step of determining the three-dimensional point cloud data of the reference image in step S510. If it matches, return to step S502, and the feature extraction process of the next frame image can be executed, or the feature extraction process of the current frame image can be executed again.
  • the ICP may be used to determine the relative pose of the point cloud of the current frame image and the point cloud of the reference image, and then determine the pose of the current frame image in the second device coordinate system.
  • the inertial measurement unit can be used to determine the posture information of the first device; in step S516, the first device can be determined based on the posture of the current frame image in the second device coordinate system and the posture information of the first device. The relative pose of the device and the second device.
  • the first device may perform augmented reality processing operations based on the relative pose relationship.
  • the first device may obtain the anchor point information configured by the second device in the scene.
  • the anchor point information may include, but is not limited to, attribute information (color, size, type, etc.), identification, position, and posture of the virtual object.
  • attribute information color, size, type, etc.
  • identification identification, position, and posture of the virtual object.
  • the first device can be adjusted to the corresponding position and display the virtual object.
  • processing operations of augmented reality may also include rendering operations on real objects.
  • the first device may also display the real object after color rendering.
  • the above-mentioned augmented reality processing method is described by taking one terminal device as an example, in a scenario, the above-mentioned augmented reality processing method may be applied to multiple terminal devices.
  • the depth information is less affected by the environment, the problem of poor relocation effect due to the influence of the surrounding environment texture, illumination, angle and other factors is overcome, and the robustness of multi-person AR relocation is improved, thereby enhancing The experience of multiplayer AR.
  • this exemplary embodiment also provides an augmented reality processing device.
  • FIG. 6 schematically shows a block diagram of an augmented reality processing apparatus according to an exemplary embodiment of the present disclosure.
  • the augmented reality processing device 6 may include a first image parameter determination module 61, a second image parameter determination module 63, a first relative pose determination module 65, and a second relative pose Determine module 67.
  • the first image parameter determination module 61 may be used to determine the image parameters of the current frame image of the first device; the second image parameter determination module 63 may be used to obtain the image parameters of the reference image determined by the second device;
  • a relative pose determination module 65 can be used to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image;
  • the second relative pose determination module 67 can be used to determine the pose of the current frame image relative to the second device;
  • the augmented reality processing apparatus by determining the pose of the current frame image relative to the second device, and combining with the posture information of the first device when the current frame image is collected, the difference between the first device and the second device is determined. Relative to the posture relationship, the relocation effect is good, and the scheme is universal and easy to implement.
  • the image parameters of the current frame image include two-dimensional feature point information and three-dimensional feature point information of the current frame image.
  • the first image parameter determination module 61 may be configured to execute: Acquire the current frame image, perform feature extraction on the current frame image, and determine the two-dimensional feature point information of the current frame image; acquire the depth information corresponding to the two-dimensional feature point information, according to the two-dimensional feature point The information and the depth information corresponding to the two-dimensional feature point information determine the three-dimensional feature point information of the current frame image.
  • the process of determining the three-dimensional feature point information of the current frame image by the first image parameter determination module 61 may be configured to perform: acquiring depth information corresponding to the current frame image collected by the depth sensing module ; Register the current frame image with the depth information corresponding to the current frame image to determine the depth information of each pixel on the current frame image; determine the corresponding two-dimensional feature point information from the depth information of each pixel on the current frame image The depth information of the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information to determine the three-dimensional feature point information of the current frame image.
  • the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image.
  • the first relative pose determination module 65 may include a first relative pose determination module 65.
  • the first relative pose determining unit 701 may be configured to execute: if the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, the iterative closest point method is used to determine the three-dimensional feature point of the current frame image. The relative pose relationship between the feature point information and the three-dimensional feature point information of the reference image to obtain the pose of the current frame image relative to the second device.
  • the first relative pose determining unit 701 may be configured to perform: before determining the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image, Determine the mismatched feature point information in the two-dimensional feature point information of the current frame image and the two-dimensional feature point information of the reference image; remove the mismatched feature point information from the three-dimensional feature point information of the current frame image to determine the current frame image
  • the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image.
  • the first relative pose determination module 65 may include a first relative pose determination module 65. Two relative pose determination unit 801.
  • the second relative pose determining unit 801 may be configured to execute: if the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, then the two-dimensional feature point information of the current frame image is matched with the two-dimensional feature point information of the reference image.
  • the three-dimensional feature point information of the reference image is correlated to obtain the point-pair information; the point-pair information is used to solve the perspective n-point problem, and the position of the current frame image relative to the second device is determined according to the three-dimensional feature point information of the current frame image and the solution result. posture.
  • the second relative pose determining unit 801 performing the process of determining the pose of the current frame image relative to the second device in combination with the solution result may include: determining the three-dimensional feature point information of the current frame image according to the solution result The relative pose relationship with the three-dimensional feature point information of the reference image; the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image determined according to the solution result is used as the initial pose input, and iterative The closest point method determines the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image to determine the pose of the current frame image relative to the second device.
  • the augmented reality processing device 9 may further include a virtual object display module 91.
  • the virtual object display module 91 may be configured to execute: obtain the anchor point information configured by the second device, so as to display the anchor point on the first device based on the relative pose relationship between the first device and the second device.
  • the virtual object corresponding to the information.
  • the example embodiments described here can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiment of the present disclosure.
  • a computing device which may be a personal computer, a server, a terminal device, or a network device, etc.
  • modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory.
  • the features and functions of two or more modules or units described above may be embodied in one module or unit.
  • the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un procédé de traitement de réalité augmentée, un dispositif de traitement de réalité augmentée, un système de traitement de réalité augmentée, un support d'enregistrement et un dispositif électronique, et se rapporte au domaine technique de la réalité augmentée. Le procédé de traitement de réalité augmentée comprend : la détermination des paramètres d'image d'une image de trame actuelle d'un premier dispositif; l'acquisition des paramètres d'image d'une image de référence déterminée par un second dispositif; la détermination d'une pose de l'image de trame actuelle par rapport au second dispositif selon les paramètres d'image de l'image de trame actuelle et les paramètres d'image de l'image de référence; et la détermination de la relation de pose relative entre le premier dispositif et le second dispositif en fonction de la pose de l'image de trame actuelle par rapport au second dispositif et des informations de pose du premier dispositif lorsque l'image de trame actuelle est acquise, de façon à effectuer une opération de traitement de réalité augmentée en utilisant la relation de pose relative entre le premier dispositif et le second dispositif. La présente invention peut améliorer l'effet de relocalisation.
PCT/CN2020/116290 2019-10-31 2020-09-18 Procédé et appareil de traitement de réalité augmentée, système, support d'enregistrement et dispositif électronique WO2021082801A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911055871.2A CN110866977B (zh) 2019-10-31 2019-10-31 增强现实处理方法及装置、系统、存储介质和电子设备
CN201911055871.2 2019-10-31

Publications (1)

Publication Number Publication Date
WO2021082801A1 true WO2021082801A1 (fr) 2021-05-06

Family

ID=69653264

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/116290 WO2021082801A1 (fr) 2019-10-31 2020-09-18 Procédé et appareil de traitement de réalité augmentée, système, support d'enregistrement et dispositif électronique

Country Status (2)

Country Link
CN (1) CN110866977B (fr)
WO (1) WO2021082801A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115937478A (zh) * 2022-12-26 2023-04-07 北京字跳网络技术有限公司 一种标定信息确定方法、装置、电子设备和存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110866977B (zh) * 2019-10-31 2023-06-16 Oppo广东移动通信有限公司 增强现实处理方法及装置、系统、存储介质和电子设备
US20240007590A1 (en) * 2020-09-30 2024-01-04 Beijing Zitiao Network Technology Co., Ltd. Image processing method and apparatus, and electronic device, and computer readable medium
CN112365530A (zh) * 2020-11-04 2021-02-12 Oppo广东移动通信有限公司 增强现实处理方法及装置、存储介质和电子设备
CN112270769B (zh) * 2020-11-11 2023-11-10 北京百度网讯科技有限公司 一种导游方法、装置、电子设备及存储介质
CN113051424A (zh) * 2021-03-26 2021-06-29 联想(北京)有限公司 一种基于slam地图的定位方法和装置

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130148851A1 (en) * 2011-12-12 2013-06-13 Canon Kabushiki Kaisha Key-frame selection for parallel tracking and mapping
CN106355647A (zh) * 2016-08-25 2017-01-25 北京暴风魔镜科技有限公司 增强现实系统和方法
CN108734736A (zh) * 2018-05-22 2018-11-02 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质
CN109949422A (zh) * 2018-10-15 2019-06-28 华为技术有限公司 用于虚拟场景的数据处理方法以及设备
CN109976523A (zh) * 2019-03-22 2019-07-05 联想(北京)有限公司 信息处理方法和电子设备
CN110264509A (zh) * 2018-04-27 2019-09-20 腾讯科技（深圳）有限公司 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN110275968A (zh) * 2019-06-26 2019-09-24 北京百度网讯科技有限公司 图像数据处理方法和装置
CN110286768A (zh) * 2019-06-27 2019-09-27 Oppo广东移动通信有限公司 虚拟物体显示方法、终端设备及计算机可读存储介质
CN110866977A (zh) * 2019-10-31 2020-03-06 Oppo广东移动通信有限公司 增强现实处理方法及装置、系统、存储介质和电子设备

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7957583B2 (en) * 2007-08-02 2011-06-07 Roboticvisiontech Llc System and method of three-dimensional pose estimation
JP5460499B2 (ja) * 2010-07-12 2014-04-02 日本放送協会 画像処理装置およびコンピュータプログラム
CN107025662B (zh) * 2016-01-29 2020-06-09 成都理想境界科技有限公司 一种实现增强现实的方法、服务器、终端及系统
CN110349213B (zh) * 2019-06-28 2023-12-12 Oppo广东移动通信有限公司 基于深度信息的位姿确定方法、装置、介质与电子设备

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130148851A1 (en) * 2011-12-12 2013-06-13 Canon Kabushiki Kaisha Key-frame selection for parallel tracking and mapping
CN106355647A (zh) * 2016-08-25 2017-01-25 北京暴风魔镜科技有限公司 增强现实系统和方法
CN110264509A (zh) * 2018-04-27 2019-09-20 腾讯科技（深圳）有限公司 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN108734736A (zh) * 2018-05-22 2018-11-02 腾讯科技（深圳）有限公司 相机姿态追踪方法、装置、设备及存储介质
CN109949422A (zh) * 2018-10-15 2019-06-28 华为技术有限公司 用于虚拟场景的数据处理方法以及设备
CN109976523A (zh) * 2019-03-22 2019-07-05 联想(北京)有限公司 信息处理方法和电子设备
CN110275968A (zh) * 2019-06-26 2019-09-24 北京百度网讯科技有限公司 图像数据处理方法和装置
CN110286768A (zh) * 2019-06-27 2019-09-27 Oppo广东移动通信有限公司 虚拟物体显示方法、终端设备及计算机可读存储介质
CN110866977A (zh) * 2019-10-31 2020-03-06 Oppo广东移动通信有限公司 增强现实处理方法及装置、系统、存储介质和电子设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115937478A (zh) * 2022-12-26 2023-04-07 北京字跳网络技术有限公司 一种标定信息确定方法、装置、电子设备和存储介质
CN115937478B (zh) * 2022-12-26 2023-11-17 北京字跳网络技术有限公司 一种标定信息确定方法、装置、电子设备和存储介质

Also Published As

Publication number Publication date
CN110866977A (zh) 2020-03-06
CN110866977B (zh) 2023-06-16

Similar Documents

Publication Publication Date Title
WO2021082801A1 (fr) Procédé et appareil de traitement de réalité augmentée, système, support d'enregistrement et dispositif électronique
US11145083B2 (en) Image-based localization
CN110457414B (zh) 离线地图处理、虚拟对象显示方法、装置、介质和设备
US10460512B2 (en) 3D skeletonization using truncated epipolar lines
US8269722B2 (en) Gesture recognition system and method thereof
EP3968131A1 (fr) Procédé, appareil et système d'interaction d'objet, support lisible par ordinateur et dispositif électronique
JP2020507850A (ja) 画像内の物体の姿の確定方法、装置、設備及び記憶媒体
CN112889091A (zh) 使用模糊特征的相机位姿估计
WO2020248900A1 (fr) Procédé et appareil de traitement de vidéo panoramique et support de stockage
CN109992111B (zh) 增强现实扩展方法和电子设备
WO2020034981A1 (fr) Procédé permettant de générer des informations codées et procédé permettant de reconnaître des informations codées
WO2022237116A1 (fr) Procédé et appareil de traitement d'image
WO2022048428A1 (fr) Procédé et appareil de commande d'un objet cible, dispositif électronique et support de stockage
CN116848556A (zh) 使用多视图细化来增强三维模型
TWI711004B (zh) 圖片處理方法和裝置
CN111192308B (zh) 图像处理方法及装置、电子设备和计算机存储介质
CN112365530A (zh) 增强现实处理方法及装置、存储介质和电子设备
CN109816791B (zh) 用于生成信息的方法和装置
CN115578515B (zh) 三维重建模型的训练方法、三维场景渲染方法及装置
US11176636B2 (en) Method of plane tracking
CN111260544B (zh) 数据处理方法及装置、电子设备和计算机存储介质
CN112750164A (zh) 轻量化定位模型的构建方法、定位方法、电子设备
WO2023279868A1 (fr) Procédé et appareil d'initialisation simultanée de localisation et de mappage et support de stockage
WO2024087927A1 (fr) Procédé et appareil de détermination de pose, support de stockage lisible par ordinateur, et dispositif électronique
WO2023284479A1 (fr) Procédé et appareil d'estimation de plan, dispositif électronique et support de stockage

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20882491

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20882491

Country of ref document: EP

Kind code of ref document: A1