TWI518634B - Augmented reality method and system - Google Patents

Description

Augmented reality method and system

This case is about an augmented reality method and system.

Through the physical house makeup, the housekeeper can see the furniture placement in the house, so that the transaction rate of the house can be improved, and to understand the preferences of consumers. Since the introduction of Augment Reality (AR), designers/vendors have uploaded design styles and virtual furniture to the Augmented Reality Development Platform, allowing consumers to see virtual and physical housing objects on mobile devices. Combination of.

Therefore, how to track the user's position and angle regardless of the user's location in the physical space and accordingly perform appropriate AR display is one of the key points in this field.

This case proposes an augmented reality method and system, which can use a variety of methods to track the position and angle of the mobile device for AR display.

The present invention relates to an augmented reality method and system for estimating an action based on the detected physical reference object and/or a plurality of environmental features and/or the angular parameter of the mobile device and the displacement parameter. The physical location of the device in physical space is compared to the physical angle for augmented reality presentation.

According to an embodiment of the present application, an augmented reality method is proposed, which is applied Augmented the real system. The augmented reality method includes: detecting an entity reference object, a plurality of environmental features, and an angle parameter and a displacement parameter of a mobile device in a physical space; tracking the physical reference object to estimate the mobile device An entity location in the physical space is associated with a physical perspective for augmented reality presentation; when the entity reference object is not tracked, the plurality of environmental features are tracked to estimate the mobile device in the physical space An entity position is compared with the entity to perform augmented reality display; and when the entity reference object is not tracked and the plurality of environmental features are not tracked, the angle parameter and the displacement parameter are estimated according to the mobile device The physical location of the mobile device in the physical space is at an angle to the entity for augmented reality presentation.

An augmented reality system is provided according to an embodiment of the present invention, comprising: a mobile device, and an augmented reality platform coupled to the mobile device. In a physical space, the mobile device detects an entity reference object, a plurality of environmental features, and an angle parameter and a displacement parameter of the mobile device. The mobile device tracks the physical reference object to estimate a physical location of the mobile device in the physical space with a physical angle for augmented reality presentation. When the mobile device does not track the physical reference object, the mobile device tracks the plurality of environmental features to estimate the physical location of the mobile device in the physical space and the entity angle for augmented reality presentation. When the mobile device cannot track the physical reference object and cannot track the plurality of environmental features, the mobile device estimates the mobile device in the physical space according to the angular parameter of the mobile device and the displacement parameter The physical location is at the perspective of the entity for augmented reality presentation.

In order to better understand the above and other aspects of the present invention, the following specific embodiments, together with the drawings, are described in detail below:

100‧‧‧Augmented Reality System

110‧‧‧ mobile devices

150‧‧‧AR platform

170‧‧‧Network

111‧‧‧Photographic unit

112‧‧‧Rotary sensor

113‧‧‧Acceleration sensor

114‧‧‧ screen

115‧‧‧AR application

121‧‧‧AR Tracking Module

122‧‧‧Environmental Feature Tracking Module

124‧‧‧Characteristic database

123‧‧‧Mobile Estimation Module

152‧‧‧AR content size module

151‧‧‧AR editing module

220A-220D‧‧‧Environmental characteristics

210‧‧‧Entity reference objects

320, 320’‧‧‧AR furniture

230‧‧‧AR furniture

310‧‧‧Entity reference object

Figure 1 shows a schematic diagram of an augmented reality system in accordance with an embodiment of the present invention.

2A to 2E are schematic diagrams showing the position and angle of the motion estimation device for performing AR display according to an embodiment of the present invention.

3A and 3B are diagrams showing the editing of an AR object in accordance with an embodiment of the present invention.

Figure 4 is a flow chart showing an augmented reality method in accordance with an embodiment of the present invention.

The technical terms of the present specification refer to the idioms in the technical field, and some of the terms are explained or defined in the specification, and the explanation of the terms is based on the description or definition of the specification. Various embodiments of the present disclosure each have one or more of the technical features. Those skilled in the art can selectively implement some or all of the technical features of any embodiment, or selectively combine some or all of the technical features of these embodiments, where possible.

Reference is now made to Fig. 1, which shows a schematic diagram of an augmented reality system in accordance with an embodiment of the present invention. As shown in FIG. 1, the augmented reality system 100 includes a mobile device 110 and an AR platform 150. The mobile device 110 and the AR platform 150 can communicate with each other through the network 170. Network 170 can include a wired/wireless network.

The mobile device 110 includes a photographing unit 111, a rotation sensor 112, an acceleration sensor 113, a screen 114, and an AR application 115. The AR application 115 is installed in a memory (not shown) of the mobile device 110. The AR application 115 includes an AR tracking module 121, an environment feature tracking module 122, a motion estimation module 123, and a feature database 124. The AR platform 150 includes an AR editing module 151 and an AR content size module 152.

The photographing unit 111 is used to photograph a physical object in a physical space. The rotation sensor 112 is configured to sense an angular parameter such as an angular acceleration and an angular momentum of the mobile device 110 to sense the angle of the mobile device 110. The rotation sensor 112 may be, for example, a gyroscope or the like. The acceleration sensor 113 is configured to sense an acceleration parameter/displacement parameter of the mobile device 110 to sense the position of the mobile device 110. The screen 114 is used to present an image, such as an image taken by the photographing unit 111, or an image of an AR display transmitted by the AR application 115.

The AR application 115 is used to present the AR presentation on the screen 114 on the fly. That is to say, the user can view the screen 114 on the same time, and the physical image captured by the photographing unit 111 (for example, a physical object (human, physical reference object, wall, etc.) in the physical space) can be simultaneously presented and The image of the AR display sent by the AR application 115.

If the mobile device 110 is moving/rotating, the AR configuration displayed on the screen 114 of the mobile device 110 will be moved/rotated (via the control of the AR application 115) to allow the user to visually feel while viewing. Close to the current situation.

When the photographing unit 111 of the mobile device 110 can capture the physical reference object, the AR tracking module 121 can calculate the physical position of the mobile device 110 in the physical space. A physical reference object is usually a flat object placed on the ground. Upon initialization (eg, when the AR application 115 is executed to detect this entity reference object for the first time), the AR application 115 can calculate the mobile device 110 in this entity through the entity reference object and (optional) other environmental characteristics. The initial physical location in space. Thereafter, if the photographing unit 111 can always capture/detect the physical reference object, the AR tracking module 121 can track the physical position and the physical angle of the mobile device 110 in the physical space.

When the photographing unit 111 of the mobile device 110 can capture environmental features in the physical space (for example, a map hanging on a wall, etc.), the environmental feature tracking module 122 can calculate/track the mobile device 110 in the physical space accordingly. The physical location in . In detail, after the AR tracking module 121 calculates the initial physical location of the mobile device 110 in the physical space, the environmental feature tracking module 122 can continuously calculate/track the mobile device 110 in the physical space according to the environmental characteristics. Physical location.

Further, the environmental feature tracking module 122 can find environmental features according to computer vision (CV). Thereafter, the environmental feature tracking module 122 can continuously establish an environmental feature map (stored in the feature database 124). If the physical reference object can be detected all the time, the environmental feature tracking module 122 repeats the above operations. However, if the physical reference object cannot be detected, the environmental feature tracking module 122 can utilize an algorithm (eg, random sampling consistent (RANdom SAmple) Consensus, RANSAC) algorithm to find these environmental features located close to the physical reference object. Thereafter, the environment feature tracking module 122 can utilize another algorithm (such as an Iterative Closest Point (ICP) algorithm) to remove the environmental features from the found location near the physical reference object. One or more environmental characteristics of noise. Thereby, the environmental feature tracking module 122 can estimate the position, angle, and the like of the mobile device 110 according to the environmental characteristics.

After the AR tracking module 121 calculates the initial physical location of the mobile device 110 in the physical space, the motion estimation module 123 can continuously track/estimate the mobile device according to the rotation sensor 112 and the acceleration sensor 113. 110 position and angle.

Further, the motion estimation module 123 can estimate the angle parameter of the mobile device 110 according to the angular velocity of the mobile device 110 detected by the rotation sensor 112; and the motion detected by the acceleration sensor 113. The speed of the device 110 is used to estimate the displacement and position parameters of the mobile device 110. Therefore, the motion estimation module 123 can continuously track/estimate the position and angle of the mobile device 110.

In the data priority order, the AR tracking module 121 is higher than the environment feature tracking module 122, and the environment feature tracking module 122 is higher than the motion estimation module 123. That is to say, in an embodiment of the present invention, if the mobile device 110 can capture the physical reference object all the time, the physical position of the mobile device 110 estimated/tracked by the AR tracking module 121 is preferentially adopted because it is accurate. The degree is higher than the other two modules 122/123. Once the mobile device 110 can't capture the physical reference object, but can capture enough environmental features, then the second best, adopt The physical location of the mobile device 110 as estimated/tracked by the environmental feature tracking module 122. If the mobile device 110 does not capture the physical reference object and the number of captured environmental features is insufficient for the environmental feature tracking module 122 to estimate/track the physical location of the mobile device 110, then retreat and then retreat The physical location of the mobile device 110 estimated/tracked by the mobile estimation module 123 is adopted.

The feature database 124 is used to store captured/detected feature points (including physical reference objects and environmental features).

The AR editing module 151 can perform editing of the AR presentation according to the data (for example, the size of the physical space, etc.) returned by the AR application 115 of the mobile device 110. For example, if the user feels that the current AR display does not meet the requirements (for example, the physical space number corresponding to the AR display does not match the number of physical spaces of the current user's physical space, or the user wants to replace the AR. The mobile device 110 can be transmitted back to the AR platform 150, and the AR editing module 151 can perform AR editing (for example, adjusting the placement position, rotation angle, size, etc. of the AR virtual object) to achieve customization.

The AR content size module 152 calculates the corresponding area of the AR configuration in the physical space. For example, if the physical area of the entity reference object is A, the AR area of the AR configuration is B, and the AR area of the AR reference object in the AR configuration is C, the physical area occupied by the AR configuration in the physical space is Y=A*. B/C. That is, when the area conversion is performed, the AR reference object is corresponding to the physical reference object to find the converted occupied area of the AR configuration.

For example, but not limited to, an object of size A0 is used as an entity reference. For the object, A = 1189 (cm) * 841 (cm). The physical reference object refers to a reference object placed in the physical space. When the AR application 115 detects the physical reference object, the physical reference object is regarded as a reference positioning point. In general, a solid reference object is placed at the center point of the physical space. For example, if the physical space is the living room, the physical reference object will be placed at the center of the living room.

The so-called AR configuration typically includes the type, size, shape, placement, etc. of the AR furniture to be placed. For example, when designing an AR configuration suitable for a living room, the AR configuration typically includes common AR furniture such as a coffee table, a TV cabinet, a sofa, and the like. When designing the AR configuration, the AR area corresponding to the AR configuration is also designed. In other words, if a house consists of a living room (5 pings), a restaurant (3 pings), a master bedroom (5 pings) and a general bedroom (4 pings), it will be designed/generated separately when designing the AR configuration. Suitable for individual AR configurations in the living/dining/room.

When designing the AR configuration, the AR reference object is also set in the AR configuration, and the preset virtual placement position is also located at the central point of the AR space to correspond to the physical reference object.

Therefore, when performing the area conversion, the AR content size module 152 corresponds the position of the AR reference object to the position of the physical reference object, and scales/places the area of the AR reference object to be the same as the physical reference object. The AR area of the AR configuration is reduced/released according to the ratio to obtain a corresponding area of the AR configuration in the physical space.

In an embodiment of the present invention, after the AR virtual object is adjusted/edited by the AR editing module 151, the AR platform 150 can transmit back to the AR of the mobile device 110. Application 115. Accordingly, the AR application 115 can obtain the adjusted AR configuration, and cannot perform 3D registration again.

It will now be described how an embodiment of the present invention tracks/estimates the position and angle of the mobile device 110. As shown in FIG. 2A, the user holds the mobile device 110 and aligns with the physical reference marker 210, whereby the AR tracking module 121 can calculate the initial physical location of the mobile device 110 in the physical space. If the mobile device 110 moves to a position where the physical reference object 210 cannot be captured, as shown in FIG. 2B, the environmental feature tracking module 122 can detect the currently captured environmental features 220A-220D to track/pre-take. The position and angle of the mobile device 110 is estimated to allow the AR application 115 to display an AR display (e.g., AR furniture 230) corresponding to the current position and angle on the screen 114, as shown in FIG. 2C.

If the mobile device 110 is moved to the inability to capture the physical reference object 210 and the environmental features 220A-220D are not captured (or the mobile device that is captured with insufficient environmental features to allow the environmental feature tracking module 122 to estimate/track) If the position of the physical position of 110 is as shown in FIG. 2D, the motion estimation module 123 can know the displacement amount of the mobile device 110 at each time point according to the rotation sensor 112 and the acceleration sensor 113. The angle, and thus the physical coordinates of the mobile device 110 at each moment, is known. Therefore, the motion estimation module 123 can also track/estimate the position and angle of the mobile device 110, so that the AR application 115 can display the AR display corresponding to the current position and angle on the screen 114, as shown in FIG. 2E. Shown.

In addition, in an embodiment of the present invention, the AR tracking module 121 and the environment The feature tracking module 122 and the motion estimation module 123 continue to operate to continuously estimate the position and angle of the mobile device 110.

In addition, if the user feels that the AR furniture in the AR configuration is not satisfactory during the AR display process (for example, the size of the AR furniture is not suitable), the user can pass the parameters back to the AR platform 150 through the AR application 115, allowing the designer to The AR editing module 151 and/or the AR content size module 152 of the AR platform 150 modify/adjust the AR furniture in the AR configuration and transmit it back to the user. For example, taking FIG. 3A as an example, when performing AR display, the AR application 115 of the mobile device 110 can measure the area of the physical space to select a suitable area and location from a plurality of pre-designed AR configurations. Like the AR style. Here, the so-called AR style refers to a combination of a plurality of AR furnitures that are pre-designed. The AR application 115 displays the AR configuration selected by the user on the screen as shown in FIG. 3A. The physical reference object 310 and the AR furniture 320 are displayed on the screen. The user feels that the size/color of the AR furniture 320 is not right/disliked, and the user can pass back to the AR platform 150 through the AR application 115. The AR platform 150 modifies the AR furniture 320 to the AR furniture 320' and transmits it back to the AR application 115 and displays it on the screen 114 as shown in FIG. 3B. In Figure 3B, the color/size of the AR furniture 320' has been modified.

Reference is now made to Fig. 4, which shows a flow chart of an augmented reality method in accordance with an embodiment of the present invention. As shown in FIG. 4, in step 410, an entity reference object, a plurality of environmental features, and an angle parameter and a displacement parameter of a mobile device are detected in a physical space. In step 420, the entity reference object is tracked to estimate a physical location of the mobile device in the physical space and a physical angle. Augmented reality display. In step 430, when the entity reference object is not tracked, the plurality of environmental features are tracked to estimate the physical location of the mobile device in the physical space and the entity angle for augmented reality presentation. In step 440, when the entity reference object is not tracked and the plurality of environmental features are not tracked, the entity of the mobile device in the physical space is estimated according to the angle parameter of the mobile device and the displacement parameter. The location is at an angle to the entity for augmented reality presentation.

In summary, in the embodiment of the present invention, since the position and angle of the mobile device can be estimated in various ways, even when the mobile device moves to detect no physical reference object (usually placed at the center of the room) The position and angle of the mobile device can still be estimated based on environmental characteristics and/or acceleration/rotation sensors, so that the AR display is not distorted by the movement of the mobile device.

In summary, although the present invention has been disclosed above by way of example, it is not intended to limit the present invention. Those who have ordinary knowledge in the technical field of the present invention can make various changes and refinements without departing from the spirit and scope of the present case. Therefore, the scope of protection of this case is subject to the definition of the scope of the patent application attached.

410~440‧‧‧Steps

Claims (12)

An augmented reality method is applied to an augmented reality system, the method comprising: detecting an entity reference object, a plurality of environmental features and an angle parameter of a mobile device in a physical space a displacement parameter; tracking the physical reference object to estimate a physical location of the mobile device in the physical space and a physical angle for augmented reality display; when the physical reference object is not tracked, tracking the multiple An environmental feature to estimate the physical location of the mobile device in the physical space and the physical perspective for augmented reality presentation; and when the physical reference object is not tracked and the plurality of environmental features are not tracked According to the angle parameter of the mobile device and the displacement parameter, the physical position of the mobile device in the physical space and the entity angle are estimated to perform augmented reality display. The augmented reality method as described in claim 1, further comprising: utilizing the initial detection of the entity reference object to calculate an initial physical location of the mobile device in the physical space; and if The entity refers to the object, and accordingly, the augmented reality display is performed by tracking the physical position and the physical angle of the mobile device in the physical space. The augmented reality method of claim 1, wherein the step of detecting the plurality of environmental features comprises: Finding the plurality of environmental features according to computer vision; continuously establishing and storing an environmental feature map; if the physical reference object cannot be detected, finding the plurality of environmental features located close to the physical reference object; Removing the one or more environmental features of the plurality of environmental features that may be noise; and estimating the physical location of the mobile device from the remaining one based on the remaining plurality of environmental features to perform amplification Reality display. The augmented reality method of claim 1, wherein the step of detecting the angle parameter and the displacement parameter of the mobile device comprises: detecting the mobile device by using a rotation sensor An angle parameter; and an acceleration sensor to detect the displacement parameter of the mobile device. The augmented reality method as described in claim 1, further comprising: returning the mobile device to an augmented reality platform for editing of the augmented reality display; and the augmented reality platform After editing, it is passed back to the mobile device for augmented reality display. The augmented reality method as described in claim 1, further comprising: an augmented reality reference object in an augmented reality configuration corresponding to the entity reference object; the augmented reality reference object The area is reduced/placed to the same size as the physical reference object; According to the ratio, an augmented reality area of the augmented reality configuration is reduced/released to obtain a corresponding area of the augmented reality configuration in the physical space. An augmented reality system includes: a mobile device, and an augmented reality platform coupled to the mobile device, wherein the mobile device detects an entity reference object and multiple environmental features in a physical space An angle parameter and a displacement parameter with the mobile device; the mobile device tracks the physical reference object to estimate a physical location of the mobile device in the physical space with a physical angle for augmented reality display; When the mobile device does not track the physical reference object, the mobile device tracks the plurality of environmental features to estimate the physical location of the mobile device in the physical space and the entity angle for augmented reality display; When the mobile device cannot track the physical reference object and cannot track the plurality of environmental features, the mobile device estimates the mobile device in the physical space according to the angular parameter of the mobile device and the displacement parameter The physical location is at the perspective of the entity for augmented reality presentation. The augmented reality system of claim 7, wherein the mobile device first detects the physical reference object to calculate an initial physical location of the mobile device in the physical space; and if the action When the device detects the physical reference object, the mobile device tracks the physical location and entity of the mobile device in the physical space. Angle, for augmented reality display. The augmented reality system of claim 7, wherein the mobile device finds the plurality of environmental features according to computer vision; the mobile device continuously establishes and stores an environmental feature map; if the mobile device cannot Detecting the physical reference object, the mobile device finds the plurality of environmental features located near the physical reference object; the mobile device removes one or more of the plurality of environmental features that may be noise An environmental feature; and based on the remaining plurality of environmental features, the mobile device estimates the physical location and the entity angle for augmented reality presentation. The augmented reality system of claim 7, wherein the mobile device uses a rotation sensor to detect the angle parameter of the mobile device; and the mobile device utilizes an acceleration sensor to detect The displacement parameter of the mobile device is measured. The augmented reality system of claim 7, wherein the mobile device is transmitted back to the augmented reality platform for editing of the augmented reality display; and the augmented reality platform is performed. After editing, the augmented reality platform is transmitted back to the mobile device for augmented reality display. The augmented reality system of claim 7, wherein: the mobile device is an augmented reality reference object pair in an augmented reality configuration The object should be referenced to the entity; the mobile device scales/amplifies the area of the augmented reality reference object to a size equal to the physical reference object; and shrinks/distributes the augmented reality configuration according to the ratio Upon augmenting the real area, the mobile device obtains the corresponding area of the augmented reality configuration in the physical space.