WO2021082801A1

WO2021082801A1 - Augmented reality processing method and apparatus, system, storage medium and electronic device

Info

Publication number: WO2021082801A1
Application number: PCT/CN2020/116290
Authority: WO
Inventors: 黄锋华
Original assignee: Oppo广东移动通信有限公司
Priority date: 2019-10-31
Filing date: 2020-09-18
Publication date: 2021-05-06
Also published as: CN110866977A; CN110866977B

Abstract

The present application provides an augmented reality processing method, an augmented reality processing device, an augmented reality processing system, a storage medium, and an electronic device, and relates to the technical field of augmented reality. The augmented reality processing method comprises: determining image parameters of a current frame image of a first device; acquiring image parameters of a reference image determined by a second device; determining a pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image; and determining the relative pose relationship between the first device and the second device according to the pose of the current frame image relative to the second device and the pose information of the first device when the current frame image is acquired, so as to perform an augmented reality processing operation by using the relative pose relationship between the first device and the second device. The present application can improve the relocation effect.

Description

Augmented reality processing method and device, system, storage medium and electronic equipment

Cross-references to related applications

This application claims the priority of the Chinese patent application filed on October 31, 2019, with the application number 201911055871.2 and titled "Augmented Reality Processing Method and Device, System, Storage Medium and Electronic Equipment", the entire content of the Chinese patent application The entire text is incorporated by reference.

Technical field

The present disclosure relates to the field of augmented reality technology, and in particular to an augmented reality processing method, an augmented reality processing device, an augmented reality processing system, a storage medium, and electronic equipment.

Background technique

Augmented Reality (AR) is a technology that integrates the virtual world and the real world. This technology has been widely used in education, games, medical care, the Internet of Things, intelligent manufacturing and other fields.

In the multi-person AR solution, the relocation effect plays a crucial role in the AR experience. However, because the shooting angles of the mapping device and the relocation device are not the same, in the process of determining the pose relationship between the mapping device and the relocation device, the problem of feature mismatch may occur, resulting in poor relocation results .

Summary of the invention

According to a first aspect of the present disclosure, there is provided an augmented reality processing method, which is applied to a first device, and the augmented reality processing method includes: determining image parameters of a current frame image of the first device; and acquiring the image parameters determined by the second device Image parameters of the reference image; determine the pose of the current frame image relative to the second device based on the image parameters of the current frame image and the image parameters of the reference image; capture the current frame image based on the pose of the current frame image relative to the second device When the posture information of the first device is used, the relative posture relationship between the first device and the second device is determined, so as to use the relative posture relationship between the first device and the second device to perform augmented reality processing operations.

According to a second aspect of the present disclosure, there is provided an augmented reality processing device applied to a first device. The augmented reality processing device includes a first image parameter determination module, a second image parameter determination module, and a first relative pose determination module And the second relative pose determination module.

Specifically, the first image parameter determination module is used to determine the image parameters of the current frame image of the first device; the second image parameter determination module is used to obtain the image parameters of the reference image determined by the second device; the first relative pose The determining module is used to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image; the second relative pose determining module is used to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image relative to the second device. The pose and the pose information of the first device when the current frame image is collected, determine the relative pose relationship between the first device and the second device, so as to use the relative pose relationship between the first device and the second device to perform augmented reality processing operations.

According to a third aspect of the present disclosure, there is provided an augmented reality processing system applied to a first device. The augmented reality processing system includes a camera module, a depth sensing module, an inertial measurement unit, a real-time positioning and map construction unit, and Augmented reality processing device.

Specifically, the camera module is used to collect the current frame image; the depth sensing module is used to collect the depth information corresponding to the current frame image; the inertial measurement unit is used to measure the inertial information of the first device; the instant positioning and map construction unit is used To obtain the current frame image and inertial information, and generate the posture information of the first device based on the inertial information; the augmented reality processing device is used to determine the image parameters of the current frame image; to obtain the image parameters of the reference image determined by the second device; The image parameters of the current frame image and the image parameters of the reference image determine the pose of the current frame image relative to the second device, and combine the posture information of the first device when the current frame image is collected to determine the relative position of the first device and the second device Posture relationship.

According to a fourth aspect of the present disclosure, there is provided a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned augmented reality processing method is implemented.

According to a fifth aspect of the present disclosure, there is provided an electronic device including a processor; and a memory for storing executable instructions of the processor; the processor is configured to execute the aforementioned augmented reality processing by executing the executable instructions method.

Description of the drawings

FIG. 1 shows a schematic diagram of a scenario architecture suitable for implementing exemplary embodiments of the present disclosure;

Fig. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present invention;

Fig. 3 schematically shows an architecture diagram of an augmented reality processing system according to an exemplary embodiment of the present disclosure;

Fig. 4 schematically shows a flowchart of an augmented reality processing method according to an exemplary embodiment of the present disclosure;

FIG. 5 schematically shows a flowchart of determining the relative pose relationship between the first device and the second device by using the iterative closest point method according to the present disclosure;

Fig. 6 schematically shows a block diagram of an augmented reality processing device according to an exemplary embodiment of the present disclosure;

FIG. 7 schematically shows a block diagram of a first relative pose determination module according to an exemplary embodiment of the present disclosure;

Fig. 8 schematically shows a block diagram of a first relative pose determination module according to another exemplary embodiment of the present disclosure;

FIG. 9 schematically shows a block diagram of an augmented reality processing apparatus according to another exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms, and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed To those skilled in the art. The described features, structures or characteristics can be combined in one or more embodiments in any suitable way. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, the well-known technical solutions are not shown or described in detail to avoid overwhelming the crowd and obscure all aspects of the present disclosure.

In addition, the drawings are only schematic illustrations of the present disclosure, and are not necessarily drawn to scale. The same reference numerals in the figures denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

The flowchart shown in the drawings is only an exemplary description, and does not necessarily include all the steps. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may be changed according to actual conditions.

Fig. 1 shows a schematic diagram of a scenario architecture suitable for implementing exemplary embodiments of the present disclosure.

As shown in FIG. 1, the augmented reality processing solution architecture of the exemplary embodiment of the present disclosure may include a first device 1001 and a second device 1002. Wherein, the second device 1002 is used to map the scene in which it is located, and the first device 1001 is a terminal device that is currently to perform an augmented reality processing operation in the scene.

The first device 1001 and the second device 1002 may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, smart wearable devices, and the like.

The first device 1001 and the second device 1002 may be communicatively connected. Specifically, the connection can be established through Bluetooth, hotspot, WiFi, mobile network, etc., so that the second device 1002 can directly transmit data with the first device 1001 without the data going through the server.

For the mapping process of the second device 1002, the video frame image can be obtained based on the camera module of the second device 1002, and the depth information corresponding to each video frame image can be obtained based on the depth sensing module. Thus, for each For frame images, the second device 1002 can determine two-dimensional feature point information and three-dimensional feature point information. In the case of using real-time positioning and mapping (Simultaneous Localization And Mapping, SLAM) for mapping, the second device 1002 can send the two-dimensional feature point information and three-dimensional feature point information of each frame of image or key frame image to the first One device 1001.

When the first device 1001 performs the augmented reality processing operation, the first device 1001 can obtain the current frame image taken by its own camera module, and based on the corresponding depth information, determine the two-dimensional feature point information and the three-dimensional feature point information of the current frame image. Feature point information. Subsequently, the first device 1001 may match the two-dimensional feature point information and the three-dimensional feature point information of the current frame image with the two-dimensional feature point information and the three-dimensional feature point information of the image determined by the second device 1002, and determine the current frame based on the matching result The pose of the image relative to the second device 1002. Next, the first device 1001 combines its own posture information to determine the relative posture relationship between the first device 1001 and the second device 1002. In this case, if the second device 1002 configures anchor point information when creating a map, the first device 1001 can obtain the anchor point information, so that the first device 1001 and the second device 1002 can be in the same position in the scene Display virtual objects and perform other augmented reality processing operations.

In addition to the direct data communication between the first device 1001 and the second device 1002, the second device 1002 can also send information to the first device 1001 by means of the server 1003. Among them, the server 1003 may be a cloud server.

Specifically, the second device 1002 may send the two-dimensional feature point information and the three-dimensional feature point information of each frame of image in the mapping process to the server 1003, and may also send the configured anchor point information to the server 1003. When the first device 1001 is in the scene created by the second device 1002, the first device 1001 can obtain the two-dimensional feature point information and the three-dimensional feature point information of each frame of image during the mapping process from the server 1003, and compare it with the current The information of the frame image is matched to determine the relative pose relationship between the first device 1001 and the second device 1002.

It should be understood that the number of terminal devices and servers in FIG. 1 is only illustrative, and any number of terminal devices and servers may be provided according to implementation needs. For example, the server 1003 may be a server cluster composed of multiple servers. In addition, multi-person AR interaction with more than three persons can also be realized.

For the convenience of the following description, the terminal device used for mapping is described as the second device, and the terminal device currently used for processing operations of augmented reality is described as the first device for distinction. It should be understood that the second device may be a terminal device currently performing processing operations in some scenarios, and the first device may also be a terminal device for mapping in some scenarios.

It should be noted that the augmented reality processing method of the exemplary embodiment of the present disclosure is executed by the first device, and accordingly, the augmented reality processing device and the augmented reality processing system described below may be configured in the first device.

Fig. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an exemplary embodiment of the present disclosure. That is to say, FIG. 2 exemplarily shows a schematic diagram of the computer structure of the above-mentioned first device.

It should be noted that the computer system 200 of the electronic device shown in FIG. 2 is only an example, and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 2, the computer system 200 includes a central processing unit (CPU) 201, which can be based on a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203 And perform various appropriate actions and processing. In RAM 203, various programs and data required for system operation are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input/output (I/O) interface 205 is also connected to the bus 204.

The following components are connected to the I/O interface 205: an input part 206 including a keyboard, a mouse, etc.; an output part 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and speakers, etc.; a storage part 208 including a hard disk, etc. ; And a communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the Internet. The drive 210 is also connected to the I/O interface 205 as needed. The removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that the computer program read from it is installed into the storage part 208 as needed.

In particular, according to an embodiment of the present disclosure, the process described below with reference to a flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication part 209, and/or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, various functions defined in the system of the present application are executed.

It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more for realizing the specified logic function. Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by It is realized by a combination of dedicated hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented in software or hardware, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

As another aspect, this application also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist alone without being assembled into the electronic device. in. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device realizes the method described in the following embodiments.

Hereinafter, the augmented reality processing system of the exemplary embodiment of the present disclosure will be described with reference to FIG. 3.

Referring to FIG. 3, the augmented reality processing system of the exemplary embodiment of the present disclosure may include an inertial measurement unit 31, a camera module 32, a depth sensing module group 33, an instant positioning and map construction unit 34, and an augmented reality processing device 35.

The inertial measurement unit 31 may include a gyroscope and an accelerometer, which respectively measure the angular velocity and acceleration of the first device, and thereby determine the inertial information of the first device.

The camera module 31 can be used to collect video frame images, where the video frame images are RGB images. During the execution of the following augmented reality processing, the camera module 31 can acquire the current frame of image for subsequent processing.

The depth sensing module 33 may be used to collect depth information. Specifically, the depth sensing module may be a dual camera module, a structured light module, or a TOF (Time-Of-Flight, Time-Of-Flight) module. This disclosure does not impose special restrictions on this.

The real-time positioning and map construction unit 34 can be used to obtain the inertial information sent by the inertial measurement unit 31 and the image sent by the camera module 32, and perform the mapping and relocation process.

The augmented reality processing device 35 can obtain the current frame image sent by the instant positioning and map construction unit 34, determine the image parameters of the current frame image, obtain the image parameters of the reference image determined by the second device, and determine the image parameters of the reference image according to the current frame image. The image parameters and the image parameters of the reference image determine the pose of the current frame image relative to the second device, and combine the pose information of the first device when the current frame image is used to determine the relative pose relationship between the first device and the second device.

Specifically, the augmented reality processing device 35 may obtain the current frame image and the corresponding posture information sent by the instant positioning and map construction unit 34. Extract the two-dimensional feature point information of the current frame image; obtain the depth information corresponding to the current frame image from the depth sensing module 33, and determine the three-dimensional feature point information of the current frame image according to the depth information corresponding to the two-dimensional feature point information; Use the two-dimensional feature point information and the three-dimensional feature point information of the current frame image and the two-dimensional feature point information and the three-dimensional feature point information of the image determined by the second device to determine the pose of the current frame image relative to the second device, and Combining the posture information of the first device, determine the relative posture relationship between the first device and the second device, so as to use the relative posture relationship between the first device and the second device to perform augmented reality processing operations.

In addition, the augmented reality processing device 35 may also include an anchor point acquisition module. The anchor point acquisition module is used to acquire the anchor point information configured by the second device, so as to display the virtual object corresponding to the anchor point information on the first device based on the relative pose relationship between the first device and the second device.

In addition, the first device can also add anchor point information in the scene so that other devices can display and perform interactive operations. In this case, the enhanced display processing system may further include an anchor point adding unit. The anchor point adding unit may be used to add anchor point information in the scene where the first device is located. Specifically, the anchor point adding unit may include an application program 36 as shown in FIG. 3, and the user holding the first device may implement the addition of anchor point information by means of the application program 36.

It should be noted that the second device involved in the exemplary embodiment of the present disclosure may also have a system architecture as shown in FIG. 3.

The augmented reality processing method of the exemplary embodiment of the present disclosure will be described below. Referring to FIG. 4, the augmented reality processing method may include the following steps:

S42. Determine the image parameter of the current frame image of the first device.

In an exemplary embodiment of the present disclosure, the image parameter of the current frame image may include two-dimensional feature point information and three-dimensional feature point information of the current frame image.

The first device can obtain the current frame image taken by the camera module, and perform feature extraction on the current frame image to determine the two-dimensional feature point information of the current frame image. Specifically, the two-dimensional feature point information of the current frame image can be extracted based on the combination of the feature extraction algorithm and the feature descriptor.

The feature extraction algorithms adopted by the exemplary embodiments of the present disclosure may include, but are not limited to, FAST feature point detection algorithm, DOG feature point detection algorithm, Harris feature point detection algorithm, SIFT feature point detection algorithm, SURF feature point detection algorithm, etc. Feature descriptors may include, but are not limited to, BRIEF feature point descriptors, BRISK feature point descriptors, FREAK feature point descriptors, and so on.

According to an embodiment of the present disclosure, the combination of the feature extraction algorithm and the feature descriptor may be the FAST feature point detection algorithm and the BRIEF feature point descriptor. According to other embodiments of the present disclosure, the combination of the feature extraction algorithm and the feature descriptor may be a DOG feature point detection algorithm and a FREAK feature point descriptor.

It should be understood that different combinations can also be used for different texture scenes. For example, for strong texture scenes, the FAST feature point detection algorithm and the BRIEF feature point descriptor can be used for feature extraction; for weak texture scenes, DOG can be used. Feature point detection algorithm and FREAK feature point descriptor are used for feature extraction.

In addition, the first device may respond to the user's operation to perform the process of acquiring the current frame image and extracting the two-dimensional feature. For example, when the user starts the AR application, the first device may respond to the AR application startup operation, turn on the camera module, obtain the current frame image captured by the camera module, and extract the two-dimensional feature point information.

When the two-dimensional feature point information of the current frame image is determined, the depth information corresponding to the two-dimensional feature point information can be combined to determine the three-dimensional feature point information of the current frame image.

Specifically, when acquiring the current frame image, the depth information corresponding to the current frame image can be acquired through the depth sensing module. The depth sensing module may be any one of a dual camera module (for example, a color camera and a telephoto camera), a structured light module, and a TOF module.

After the current frame image and the corresponding depth information are obtained, the current frame image and the depth information can be registered to determine the depth information of each pixel on the current frame image.

For the registration process, the internal and external parameters of the camera module and the depth sensing module need to be calibrated in advance.

Specifically, a three-dimensional vector p_ir=(x, y, z) can be constructed, where x, y represent the pixel coordinates of a pixel, and z represents the depth value of the pixel. The internal parameter matrix of the depth sensing module can be used to obtain the coordinate P_ir of the pixel point in the depth sensing module coordinate system. Then, P_ir can be multiplied by a rotation matrix R, and a translation vector T is added to convert P_ir to the coordinate system of the RGB camera to obtain P_rgb. Subsequently, P_rgb can be multiplied with the internal parameter matrix H_rgb of the camera module to obtain p_rgb, p_rgb is also a three-dimensional vector, denoted as (x0, y0, z0), where x0 and y0 are the pixels of the pixel in the RGB image Coordinates, extract the pixel value of the pixel, and match it with the corresponding depth information. As a result, the alignment of the two-dimensional image information and depth information of one pixel is completed. In this case, the above process is performed for each pixel to complete the registration process.

After determining the depth information of each pixel on the current frame image, the depth information corresponding to the two-dimensional feature point information can be determined from it, and the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information can be combined to determine Get the three-dimensional feature point information of the current frame image.

In addition, after acquiring the depth information from the depth sensing module, the depth information can also be denoised to remove obviously wrong depth values in the depth information. For example, a deep neural network may be used to remove noise in the TOF image, which is not particularly limited in this exemplary embodiment.

S44. Acquire the image parameters of the reference image determined by the second device.

In an exemplary embodiment of the present disclosure, the image parameters of the reference image may include two-dimensional feature point information and three-dimensional feature point information of the reference image.

In the process of building a map, the second device can generate two-dimensional feature point information and three-dimensional feature point information of each frame of image or key frame image. When the first device is in the scene created by the second device and performs an augmented reality processing operation, the two-dimensional feature point information and the three-dimensional feature point information of these images can be acquired. It should be understood that the reference image described in the present disclosure is each frame image or key frame image generated by the second device in the mapping.

For the process of obtaining the image parameters of the reference image, in one embodiment, the second device can send the two-dimensional feature point information and three-dimensional feature point information of the reference image to the first device through Bluetooth, hotspot, WiFi, mobile network, etc. . In another embodiment, the second device may send the two-dimensional feature point information and the three-dimensional feature point information of the reference image to the cloud server, so that the first device can obtain the two-dimensional feature point information and the three-dimensional feature point information of the reference image from the cloud server. Three-dimensional feature point information.

In addition, the execution order of the above step S42 and step S44 can be interchanged, that is, the solution of the present disclosure may also execute step S44 first, and then execute step S42.

S46. Determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image.

After determining the image parameters of the current frame image and the image parameters of the reference image, the pose of the current frame image relative to the second device can be determined, that is, the pose of the current frame image in the second device coordinate system can be determined . For this process, the present disclosure provides three implementation manners, which will be described one by one below.

According to an embodiment of the present disclosure, the relationship between the two-dimensional feature point information of the current frame image and the two-dimensional feature point information of the reference image can be determined through feature matching or descriptor matching. If the two-dimensional feature point information of the current frame image is determined If the feature point information matches the two-dimensional feature point information of the reference image, the Iterative Closest Point (ICP) method can be used to determine the relative relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image. Posture relationship.

Specifically, the three-dimensional feature point information of the current frame image is the point cloud information corresponding to the current frame image, and the three-dimensional feature point information of the reference image is the point cloud information of the reference image. The two point cloud information can be used as input, and by inputting the specified pose as the initial value, the optimal relative pose after the alignment of the two point clouds is obtained by the iterative closest point method, that is, the three-dimensional feature of the current frame image is determined The relative pose relationship between the point information and the three-dimensional feature point information of the reference image. Thus, based on the posture information of the second device when acquiring the reference image, the posture of the current frame image relative to the second device can be determined.

It should be understood that before performing point cloud matching, the relationship between the two-dimensional information is determined first. Due to the determination of the two-dimensional information relationship, the method of feature matching or descriptor matching is usually adopted, and the process is simple. As a result, the entire matching process can be accelerated, and the accuracy can be improved, while the effect of troubleshooting in advance can also be achieved.

In addition, in the above-mentioned matching process of the two-dimensional feature point information, there may be a problem of mismatch due to the problem of the feature and the descriptor. Thus, the exemplary embodiment of the present disclosure may also include a solution for removing mismatched points.

The RANSAC (Random Sample Consensus) method can be used to eliminate mismatched feature point information. Specifically, a certain number of matching pairs (for example, 7 pairs, 8 pairs, etc.) are randomly selected from the matching pairs between the two-dimensional feature points of the current frame image and the two-dimensional feature points of the reference image, and the selected matching pairs are calculated The basic matrix or essential matrix between the current frame image and the reference image is based on the epipolar constraint. If a two-dimensional feature point is far from the corresponding epipolar line, for example, greater than a threshold, the two-dimensional feature can be considered The points are mismatched points. By iterating a certain number of random sampling processes, the random sampling result with the largest number of internal points is selected as the final matching result. On this basis, the mismatched feature point information can be eliminated from the three-dimensional feature point information of the current frame image.

Thus, the three-dimensional feature point information from which the mismatched feature point information is eliminated can be used to determine the pose of the current frame image relative to the second device.

According to another embodiment of the present disclosure, first, if the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, then the two-dimensional feature point information of the current frame image is matched with the three-dimensional feature point information of the reference image. Information association to get point-to-point information. Next, the point pair information can be used as input to solve the Perspective-n-Point (PnP) problem. According to the three-dimensional feature point information of the current frame image and the solution result, determine the current frame image relative to the second device Posture.

Among them, PnP is a classic method in the field of machine vision, which can determine the relative pose between the camera and the object according to n feature points on the object. Specifically, the rotation matrix and translation vector between the camera and the object can be determined according to the n feature points on the object. In addition, n may be determined to be 4 or more, for example.

According to another embodiment of the present disclosure, the relative pose relationship between the three-dimensional feature point information of the reference image and the three-dimensional feature point information obtained by combining the PnP solution result of the previous embodiment can be used as the iterative initial pose input. The point method determines the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image to determine the pose of the current frame image relative to the second device. It is easy to see that in this embodiment, PnP is combined with ICP to improve the accuracy of determining the pose relationship.

S48. Determine the relative pose relationship between the first device and the second device according to the pose of the current frame image relative to the second device and the pose information of the first device when the current frame image is collected, so as to use the first device and the second device The relative pose relationship of the two is used for augmented reality processing operations.

The inertial measurement unit of the first device can obtain the inertial information of the first device, and thereby, the 6DOF (6 Degrees Of Freedom) attitude information of the first device can be obtained. Based on the posture information of the first device and the posture of the current frame image relative to the second device determined in step S46, the relative posture relationship between the first device and the second device can be obtained.

Referring to FIG. 5, the process of determining the relative pose relationship between the first device and the second device using the ICP method will be described.

In step S502, the first device may extract the two-dimensional feature point information of the current frame image. For a weak texture scene, the DOG feature point detection algorithm and the FREAK feature point descriptor may be used for feature extraction.

In step S504, the first device can obtain the depth information input by the TOF module; in step S506, the two-dimensional feature point information can be registered with the depth information to obtain the point cloud data of the current frame image.

In step S508, the first device may determine whether the two-dimensional feature point information determined in step S502 matches the two-dimensional feature of the reference image, and if it matches, it executes the step of determining the three-dimensional point cloud data of the reference image in step S510. If it matches, return to step S502, and the feature extraction process of the next frame image can be executed, or the feature extraction process of the current frame image can be executed again.

In step S512, the ICP may be used to determine the relative pose of the point cloud of the current frame image and the point cloud of the reference image, and then determine the pose of the current frame image in the second device coordinate system.

In step S514, the inertial measurement unit can be used to determine the posture information of the first device; in step S516, the first device can be determined based on the posture of the current frame image in the second device coordinate system and the posture information of the first device. The relative pose of the device and the second device.

After the relative poses of the first device and the second device are determined, the first device may perform augmented reality processing operations based on the relative pose relationship.

For example, the first device may obtain the anchor point information configured by the second device in the scene. The anchor point information may include, but is not limited to, attribute information (color, size, type, etc.), identification, position, and posture of the virtual object. Thus, the first device can be adjusted to the corresponding position and display the virtual object.

In addition, the processing operations of augmented reality may also include rendering operations on real objects. For example, after the second device performs color rendering on a real object, the first device may also display the real object after color rendering.

It should be understood that although the above-mentioned augmented reality processing method is described by taking one terminal device as an example, in a scenario, the above-mentioned augmented reality processing method may be applied to multiple terminal devices. In view of the fact that the depth information is less affected by the environment, the problem of poor relocation effect due to the influence of the surrounding environment texture, illumination, angle and other factors is overcome, and the robustness of multi-person AR relocation is improved, thereby enhancing The experience of multiplayer AR.

It should be noted that although the various steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that these steps must be performed in the specific order, or that all the steps shown must be performed to achieve the desired the result of. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution, etc.

Further, this exemplary embodiment also provides an augmented reality processing device.

FIG. 6 schematically shows a block diagram of an augmented reality processing apparatus according to an exemplary embodiment of the present disclosure. 6, the augmented reality processing device 6 according to an exemplary embodiment of the present disclosure may include a first image parameter determination module 61, a second image parameter determination module 63, a first relative pose determination module 65, and a second relative pose Determine module 67.

Specifically, the first image parameter determination module 61 may be used to determine the image parameters of the current frame image of the first device; the second image parameter determination module 63 may be used to obtain the image parameters of the reference image determined by the second device; A relative pose determination module 65 can be used to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image; the second relative pose determination module 67 can be used to determine the pose of the current frame image relative to the second device; The pose of the frame image relative to the second device and the pose information of the first device when the current frame image is collected, determine the relative pose relationship between the first device and the second device, so as to use the relative pose of the first device and the second device Relations to perform augmented reality processing operations.

Using the augmented reality processing apparatus of the exemplary embodiment of the present disclosure, by determining the pose of the current frame image relative to the second device, and combining with the posture information of the first device when the current frame image is collected, the difference between the first device and the second device is determined. Relative to the posture relationship, the relocation effect is good, and the scheme is universal and easy to implement.

According to an exemplary embodiment of the present disclosure, the image parameters of the current frame image include two-dimensional feature point information and three-dimensional feature point information of the current frame image. In this case, the first image parameter determination module 61 may be configured to execute: Acquire the current frame image, perform feature extraction on the current frame image, and determine the two-dimensional feature point information of the current frame image; acquire the depth information corresponding to the two-dimensional feature point information, according to the two-dimensional feature point The information and the depth information corresponding to the two-dimensional feature point information determine the three-dimensional feature point information of the current frame image.

According to an exemplary embodiment of the present disclosure, the process of determining the three-dimensional feature point information of the current frame image by the first image parameter determination module 61 may be configured to perform: acquiring depth information corresponding to the current frame image collected by the depth sensing module ; Register the current frame image with the depth information corresponding to the current frame image to determine the depth information of each pixel on the current frame image; determine the corresponding two-dimensional feature point information from the depth information of each pixel on the current frame image The depth information of the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information to determine the three-dimensional feature point information of the current frame image.

According to an exemplary embodiment of the present disclosure, the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image. In this case, referring to FIG. 7, the first relative pose determination module 65 may include a first relative pose determination module 65. A relative pose determination unit 701.

Specifically, the first relative pose determining unit 701 may be configured to execute: if the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, the iterative closest point method is used to determine the three-dimensional feature point of the current frame image. The relative pose relationship between the feature point information and the three-dimensional feature point information of the reference image to obtain the pose of the current frame image relative to the second device.

According to an exemplary embodiment of the present disclosure, the first relative pose determining unit 701 may be configured to perform: before determining the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image, Determine the mismatched feature point information in the two-dimensional feature point information of the current frame image and the two-dimensional feature point information of the reference image; remove the mismatched feature point information from the three-dimensional feature point information of the current frame image to determine the current frame image The relative pose relationship between the three-dimensional feature point information after excluding the mismatched feature point information and the three-dimensional feature point information of the reference image after excluding the mismatched feature point information.

According to an exemplary embodiment of the present disclosure, the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image. In this case, referring to FIG. 8, the first relative pose determination module 65 may include a first relative pose determination module 65. Two relative pose determination unit 801.

Specifically, the second relative pose determining unit 801 may be configured to execute: if the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, then the two-dimensional feature point information of the current frame image is matched with the two-dimensional feature point information of the reference image. The three-dimensional feature point information of the reference image is correlated to obtain the point-pair information; the point-pair information is used to solve the perspective n-point problem, and the position of the current frame image relative to the second device is determined according to the three-dimensional feature point information of the current frame image and the solution result. posture.

According to an exemplary embodiment of the present disclosure, the second relative pose determining unit 801 performing the process of determining the pose of the current frame image relative to the second device in combination with the solution result may include: determining the three-dimensional feature point information of the current frame image according to the solution result The relative pose relationship with the three-dimensional feature point information of the reference image; the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image determined according to the solution result is used as the initial pose input, and iterative The closest point method determines the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image to determine the pose of the current frame image relative to the second device.

According to an exemplary embodiment of the present disclosure, referring to FIG. 9, compared to the augmented reality processing device 6, the augmented reality processing device 9 may further include a virtual object display module 91.

Specifically, the virtual object display module 91 may be configured to execute: obtain the anchor point information configured by the second device, so as to display the anchor point on the first device based on the relative pose relationship between the first device and the second device. The virtual object corresponding to the information.

Since each functional module of the augmented reality processing device in the embodiment of the present invention is the same as in the above-mentioned method and invention embodiment, it will not be repeated here.

Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described here can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiment of the present disclosure.

In addition, the above-mentioned drawings are merely schematic illustrations of the processing included in the method according to the exemplary embodiment of the present invention, and are not intended for limitation. It is easy to understand that the processing shown in the above drawings does not indicate or limit the time sequence of these processings. In addition, it is easy to understand that these processes can be executed synchronously or asynchronously in multiple modules, for example.

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field that are not disclosed in the present disclosure. . The description and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

It should be understood that the present disclosure is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

An augmented reality processing method, applied to a first device, includes:

Determining the image parameters of the current frame image of the first device;

Acquiring the image parameters of the reference image determined by the second device;

Determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image;

Determine the relative pose relationship between the first device and the second device according to the pose of the current frame image relative to the second device and the pose information of the first device when the current frame image is collected , So as to use the relative pose relationship between the first device and the second device to perform augmented reality processing operations.
The augmented reality processing method according to claim 1, wherein the image parameters of the current frame image include two-dimensional feature point information and three-dimensional feature point information of the current frame image; wherein the current frame of the first device is determined The image parameters of the frame image include:

Acquiring the current frame image, performing feature extraction on the current frame image, and determining the two-dimensional feature point information of the current frame image;

The depth information corresponding to the two-dimensional feature point information is acquired, and the three-dimensional feature point information of the current frame image is determined according to the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information.
The augmented reality processing method according to claim 2, wherein the depth information corresponding to the two-dimensional feature point information is acquired, and the depth information corresponding to the two-dimensional feature point information and the two-dimensional feature point information is determined to be the The three-dimensional feature point information of the current frame image includes:

Acquiring the depth information corresponding to the current frame image collected by the depth sensing module;

Register the current frame image with the depth information corresponding to the current frame image, and determine the depth information of each pixel on the current frame image;

Determining the depth information corresponding to the two-dimensional feature point information from the depth information of each pixel on the current frame image;

Using the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information, the three-dimensional feature point information of the current frame image is determined.
The augmented reality processing method according to claim 3, wherein the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image; wherein, according to the image parameters of the current frame image and The image parameter of the reference image to determine the pose of the current frame image relative to the second device includes:

If the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, the iterative closest point method is used to determine the three-dimensional feature point information of the current frame image and the three-dimensional feature of the reference image The relative pose relationship between the point information to obtain the pose of the current frame image relative to the second device.
The augmented reality processing method according to claim 4, wherein before determining the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image, the augmented reality processing Methods also include:

Determining mismatched feature point information in the two-dimensional feature point information of the current frame image and the two-dimensional feature point information of the reference image;

The mismatched feature point information is removed from the three-dimensional feature point information of the current frame image, so as to determine the removal of the three-dimensional feature point information after the mismatched feature point information of the current frame image and the removal of the reference image The relative pose relationship between the three-dimensional feature point information after the mismatched feature point information.
The augmented reality processing method according to claim 3, wherein the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image; wherein, according to the image parameters of the current frame image and The image parameter of the reference image to determine the pose of the current frame image relative to the second device includes:

If the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, then the two-dimensional feature point information of the current frame image is associated with the three-dimensional feature point information of the reference image, Get point-to-point information;

The point-pair information is used to solve the perspective n-point problem, and the pose of the current frame image relative to the second device is determined according to the three-dimensional feature point information of the current frame image and the solution result.
The augmented reality processing method according to claim 6, wherein determining the pose of the current frame image relative to the second device in combination with the solution result comprises:

Determining the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image according to the solution result;

The relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image determined according to the solution result is used as the initial pose input, and the current frame is determined by the iterative closest point method The relative pose relationship between the three-dimensional feature point information of the image and the three-dimensional feature point information of the reference image is used to determine the pose of the current frame image relative to the second device.
The augmented reality processing method according to any one of claims 1 to 7, wherein using the relative pose relationship between the first device and the second device to perform augmented reality processing operations comprises:

Acquire the anchor point information configured by the second device, so that based on the relative pose relationship between the first device and the second device, the virtual information corresponding to the anchor point information is displayed on the first device. Object.
An augmented reality processing device applied to a first device, including:

The first image parameter determination module is configured to determine the image parameters of the current frame image of the first device;

The second image parameter determination module is configured to obtain the image parameters of the reference image determined by the second device;

The first relative pose determining module is configured to determine the pose of the current frame image relative to the second device according to the image parameters of the current frame image and the image parameters of the reference image;

The second relative pose determining module is configured to determine the first device based on the pose of the current frame image relative to the second device and the pose information of the first device when the current frame image is collected The relative pose relationship with the second device, so as to use the relative pose relationship between the first device and the second device to perform an augmented reality processing operation.
The augmented reality processing device according to claim 9, wherein the image parameters of the current frame image include two-dimensional feature point information and three-dimensional feature point information of the current frame image, and the first image parameter determination module is configured To obtain the current frame image, perform feature extraction on the current frame image, and determine the two-dimensional feature point information of the current frame image; obtain the depth information corresponding to the two-dimensional feature point information, according to the two-dimensional The feature point information and the depth information corresponding to the two-dimensional feature point information determine the three-dimensional feature point information of the current frame image.
The augmented reality processing device according to claim 10, wherein the process of determining the three-dimensional feature point information of the current frame image by the first image parameter determination module is configured to: The depth information corresponding to the current frame image; register the current frame image with the depth information corresponding to the current frame image to determine the depth information of each pixel on the current frame image; from the current frame image The depth information corresponding to the two-dimensional feature point information is determined from the depth information of each pixel; the current frame image is determined by using the two-dimensional feature point information and the depth information corresponding to the two-dimensional feature point information The three-dimensional feature point information.
The augmented reality processing device according to claim 11, wherein the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image, and the first relative pose determination module comprises:

The first relative pose determining unit is configured to, if the two-dimensional feature point information of the current frame image matches the two-dimensional feature point information of the reference image, determine the three-dimensional image of the current frame image by using an iterative closest point method. The relative pose relationship between the feature point information and the three-dimensional feature point information of the reference image to obtain the pose of the current frame image relative to the second device.
The augmented reality processing device according to claim 12, wherein the first relative pose determining unit is further configured to: determine between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image Prior to the relative pose relationship, determine the mismatched feature point information in the two-dimensional feature point information of the current frame image and the two-dimensional feature point information of the reference image; from the three-dimensional feature point information of the current frame image The mismatched feature point information is eliminated to determine the three-dimensional feature point information of the current frame image after the mismatched feature point information is eliminated and the three-dimensional feature point of the reference image after the mismatched feature point information has been eliminated The relative pose relationship between information.
The augmented reality processing device according to claim 11, wherein the image parameters of the reference image include two-dimensional feature point information and three-dimensional feature point information of the reference image; the first relative pose determination module includes:

The second relative pose determining unit is configured to compare the two-dimensional feature point information of the current frame image with the two-dimensional feature point information of the reference image if the two-dimensional feature point information of the current frame image matches The three-dimensional feature point information of the reference image is correlated to obtain point-pair information; the point-pair information is used to solve the perspective n-point problem, and the three-dimensional feature point information of the current frame image is combined with the solution result to determine the relative value of the current frame image The pose of the second device.
The augmented reality processing device according to claim 14, wherein the second relative pose determining unit performs the process of determining the pose of the current frame image relative to the second device in combination with the solution result, and is configured to: according to the solution result Determine the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image; compare the three-dimensional feature point information of the current frame image determined according to the solution result with the reference The relative pose relationship of the three-dimensional feature point information of the image is used as the initial pose input, and the relative pose relationship between the three-dimensional feature point information of the current frame image and the three-dimensional feature point information of the reference image is determined by the iterative closest point method To determine the pose of the current frame image relative to the second device.
An augmented reality processing system applied to a first device, including:

The camera module is configured to collect the current frame image;

A depth sensing module configured to collect depth information corresponding to the current frame image;

An inertial measurement unit configured to measure the inertial information of the first device;

An instant positioning and map construction unit configured to obtain the current frame image and the inertial information, and generate posture information of the first device based on the inertial information;

The augmented reality processing device is configured to determine the image parameters of the current frame image; obtain the image parameters of the reference image determined by the second device; according to the image parameters of the current frame image and the image parameters of the reference image, Determine the pose of the current frame image relative to the second device, and determine the relative position of the first device and the second device in combination with the pose information of the first device when the current frame image is collected Posture relationship.
The augmented reality processing system according to claim 16, wherein the augmented reality processing device comprises:

The anchor point acquisition module is configured to acquire the anchor point information configured by the second device, so that based on the relative pose relationship between the first device and the second device, display on the first device and The virtual object corresponding to the anchor point information.
The augmented reality processing system according to claim 16 or 17, the augmented reality processing system further comprising:

The anchor point adding unit is configured to add anchor point information in the scene where the first device is located.
A storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the augmented reality processing method according to any one of claims 1 to 8 is realized.
An electronic device including:

Processor; and

A memory configured to store executable instructions of the processor;

Wherein, the processor is configured to execute the augmented reality processing method according to any one of claims 1 to 8 by executing the executable instructions.