TW201928779A - Augmented reality application generation system and method - Google Patents

Abstract

An augmented reality application generation system and method are disclosed. The method comprises: capturing at least one image signal, identifying and tracking an image or a mark of the image signal to obtain image or mark position information, identifying and tracking a user's posture in the image signal to obtain posture type information or posture position information, analyzing the image signal and the posture type information or the posture position information to generate a corresponding human-machine interaction information, providing at least one editing component based on the human-computer interaction information to edit interactive content of an augmented reality application, and displaying the interactive content of the augmented reality application at a real image by a superimposed manner.

Description

Augmented reality application generation system and method

The present invention relates to an augmented reality (AR) technology, and particularly to an augmented reality application generation system and method.

Augmented Reality (AR) is a way to observe the real environment by combining physical and virtual technologies, and augmented reality technology has been widely used in video games, environmental navigation, and commercial applications.

At the same time, augmented reality can use image recognition technology to detect and track real objects in the image, and use three-dimensional (3D) technology to combine preset virtual objects with real objects to display on the screen.

However, the augmented reality of the conventional technology can only display recorded multimedia objects, but the image objects on the screen can only be viewed by users or viewers, but the content of the image objects or augmented reality applications cannot be performed. Interaction or editing.

Therefore, how to solve the shortcomings of the conventional techniques has become a major issue for those skilled in the art.

The invention provides an augmented reality application generation system and method, which can For users to interact or edit the content of augmented reality applications.

The augmented reality application generation system in the present invention includes: a signal acquisition module that captures at least one image signal; and an image recognition and tracking module that recognizes and captures the signals captured by the signal acquisition module. The image or mark of the image signal to obtain the image or mark position information of the image signal; a gesture recognition tracking module that recognizes and tracks the user's posture in the image signal captured by the signal acquisition module to obtain the user Attitude type information or attitude position information; a human-machine interactive analysis module that analyzes the image signal captured by the signal acquisition module and the user's attitude type information or attitude position information obtained by the attitude recognition tracking module, Generate corresponding human-computer interaction information; an augmented reality editing module that provides at least one editing element based on the human-machine interaction information of the human-machine interaction analysis module for users to edit an augmented reality with the editing element Interactive content of the application; and an augmented reality display module that displays the interactive content of the augmented reality application edited by the user with the editing element in a superimposed manner on the real image On.

The method for generating augmented reality application in the present invention includes: capturing at least one image signal; identifying and tracking the image or mark of the image signal to obtain the image or mark position information of the image signal; identifying and tracking the image signal for use To obtain the user's attitude type information or attitude position information; analyze the image signal and the user's attitude type information or attitude position information to generate corresponding human-machine interaction information; provide at least one edit based on the human-machine interaction information A component for the user to edit an interactive content of the augmented reality application with an editing component; and an overlay to display the interactive content of the augmented reality application edited by the user with the editing component on the reality image.

In order to make the above features and advantages of the present invention more comprehensible, embodiments are described below in detail with reference to the accompanying drawings. Additional features and advantages of the present invention will be partially explained in the following description, and these features and advantages will be partially obvious from the description, or may be learned through practice of the present invention. The features and advantages of the invention are realized and achieved by means of elements and combinations specifically pointed out in the scope of the patent application. It should be understood that both the foregoing general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the scope of the invention as claimed.

1‧‧‧ Augmented Reality Application Generation System

10‧‧‧Signal Acquisition Module

11‧‧‧ Video Capture Unit

12‧‧‧ Audio Capture Unit

20‧‧‧Image recognition tracking module

30‧‧‧Attitude recognition tracking module

40‧‧‧Human-computer interaction analysis module

50‧‧‧Space positioning module

60‧‧‧ Augmented Reality Editing Module

70‧‧‧ augmented reality display module

80‧‧‧ Information Storage Module

A‧‧‧User

A1‧‧‧ Gesture

A2‧‧‧Voice

B‧‧‧ Smart Glasses Device

C‧‧‧ table

D‧‧‧ icon

E‧‧‧Editing element

F‧‧‧ Object

Steps S1 to S8‧‧‧‧

Figure 1 shows the architecture of the augmented reality application generation system in the present invention; Figure 2 shows the flowchart of the augmented reality application generation method in the present invention; Figures 3A to 3C illustrate the present invention. One embodiment of the augmented reality application generation system and method; and FIGS. 4A to 4C are schematic diagrams of another embodiment of the augmented reality application generation system and method of the present invention.

The following describes the embodiments of the present invention with specific specific implementation forms. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this description, and can also be implemented by other different specific implementation forms. Or apply.

FIG. 1 shows the architecture diagram of the augmented reality application generation system 1 in the present invention (please refer to FIGS. 3A to 4C together). As shown in Figure 1, amplification The real-world application generation system 1 includes a signal acquisition module 10, an image recognition tracking module 20, a gesture recognition tracking module 30, a human-computer interaction analysis module 40, a spatial positioning module 50, and an amplification The reality editing module 60, an augmented reality display module 70, and / or an information storage module 80.

The signal acquisition module 10 can capture at least external signals including the image signal in the current environment or space of the user A as shown in FIGS. 3A to 4C, and the signal acquisition module 10 can be connected or integrated with the user A Wear smart glasses device B. The external signal may include the image signal (environment image) or depth information of user A's current environment or space, or the image signal (such as gesture or gesture A1) or sound signal (such as voice A2) of user A. At the same time, the signal capture module 10 may have a video capture unit 11 and an audio capture unit 12, the video capture unit 11 may capture an image signal and its depth information, and the audio capture unit 12 may capture an external signal Voice signal.

The image recognition and tracking module 20 can identify and track the image or mark of the image signal of the signal capture module 10 (or the video capture unit 11) to obtain the image or mark position information of the image signal, where the image or The marked image feature point information can be specific or predefined image feature point information.

The image recognition and tracking module 20 can identify and track image feature point information of an image or a mark through one of the following algorithms: a Scale-Invariant Feature Transform (SIFT) algorithm, and accelerated robust features ( Speeded Up Robust Features (SURF) algorithm, Features from Accelerated Segment Test (FAST) algorithm, Binary Robust Independent Elementary Features (BRIEF) algorithm, or formula Oriented Brief (ORiented BRIEF; ORB) algorithm.

The attitude recognition and tracking module 30 can recognize and track the attitude of the user A in the image signal captured by the signal capture module 10 (video capture unit 11) (such as the user A of Figures 3A to 4C). Gesture A1) to obtain posture type information or posture position information of user A. The posture of the user A may be a specific posture of the user A, such as a gesture of the user A's hand grasping, fisting, or palm opening.

The human-computer interaction analysis module 40 can analyze external signals (such as image signals or depth information or sound signals) captured by the signal acquisition module 10 and the user A's posture type acquired by the posture recognition tracking module 30 Information or attitude and position information to generate corresponding human-computer interaction information. For example, the human-machine interaction analysis module 40 can analyze various human-machine interaction meanings. By analyzing external signal or gesture type information, the user A understands the interactive meaning expressed by the user A and generates corresponding human-machine interaction information.

The human-computer interaction information can know the interaction meaning between user A and the virtual editing element E. Specifically, for example, it can be known that the virtual teddy bear model of an editing element E is being grasped by the user, and the virtual toy of an editing element E The bear model has been released by user A, the virtual teddy bear model of an editing element E has been clicked by user A, or a key in the virtual keyboard of an editing element has been clicked by user A.

The spatial positioning module 50 can respectively generate corresponding three-dimensional spatial positioning information according to the image or mark position information of the image signal or the posture position information of the user A, and the three-dimensional spatial positioning information can be represented or stored in a matrix data manner.

The augmented reality editing module 60 may provide at least one editing element E as shown in FIG. 3B or 4B according to the human-machine interaction information of the human-machine interaction analysis module 40, for the user A to edit one with the editing element E. Interactive content for augmented reality applications.

Specifically, the augmented reality editing module 60 can provide a user to select different editing elements E through multiple human-machine interactions, and edit the visual presentation and interaction of the editing elements E. The editing element E can be two-dimensional (2D) images, three-dimensional (3D) models, movies or sound signals. Meanwhile, the augmented reality editing module 60 editing the interactive content of the augmented reality application may include: setting, adjusting, or combining the position, order, size, angle, or control method of the editing element E. In addition, the editing element E may provide a virtual keyboard for the user A to input information (such as text, symbols, numbers, patterns, etc.) through the virtual keyboard.

The augmented reality display module 70 (such as a screen or a smart glasses device B that can be connected or integrated with user A) can superimpose the augmented reality application edited by user A with the editing element E The interactive content is displayed on the current reality image of user A. In other words, the augmented reality display module 70 can display the augmented reality image in a superimposed manner, and superimpose and display the interactive content of the augmented reality application edited by the user A in accordance with different three-dimensional spatial positioning information. Person A on the current reality image.

The information storage module 80 may record or store the interactive content of the augmented reality application that has been edited by the augmented reality editing module 60 and displayed by the augmented reality display module 70.

FIG. 2 is a flowchart illustrating a method for generating an augmented reality application in the present invention FIG. 3 is applicable to the smart glasses device B worn by the user A as shown in FIGS. 3A to 4C. At the same time, the main technical content of the augmented reality application generation method in FIG. 2 is as follows, and the remaining technical contents are the same as those in the augmented reality application generation system in FIG. 1 described above, and will not be repeated here.

In step S1 of FIG. 2, the signal acquisition module 10 of one of the images in FIG. 1 captures an external signal including at least an image signal (environment image) of the user at the moment, and can also capture depth information or external signal of the image signal Voice signal.

In step S2 of FIG. 2, the image recognition and tracking module 20 of one of FIG. 1 recognizes and tracks the image or mark of the image signal to obtain the image or mark position information of the image signal. For example, one of the following algorithms can identify and track image feature point information of an image or mark: a scale invariant feature conversion (SIFT) algorithm, an accelerated robust feature (SURF) algorithm, an accelerated segmented feature test ( (FAST) algorithm, binary robust independent foundation feature (BRIEF) algorithm, or directional BRIEF (ORB) algorithm.

In step S3 of FIG. 2, the user A in the image signal captured by the gesture recognition and tracking module 30 in FIG. 1 and the tracking signal acquisition module 10 is identified (see FIGS. 3A to 4C). Gesture (such as gestures) to obtain the attitude type information or attitude position information of user A.

In step S4 in FIG. 2, the spatial positioning module 50 in one of the first drawings generates corresponding three-dimensional spatial positioning information according to the image or mark position information of the image signal, or the pose position information of the user A, and Spatial positioning information can be represented or stored as matrix data.

In step S5 in FIG. 2, the human-machine interaction analysis module 40 in FIG. 1 analyzes the image signal (external signal) captured by the signal acquisition module 10. No.), and the attitude type information or attitude position information of the user A obtained by the attitude recognition and tracking module 30 to generate corresponding human-machine interaction information.

In step S6 of FIG. 2, the augmented reality editing module 60 provides one or more virtual editing elements E (see FIG. 3A to (Figure 4C), for user A to edit interactive content of an augmented reality application with editing element E.

For example, the augmented reality editing module 60 may provide the user A to select different editing elements E through multiple human-machine interactions, and edit the visual presentation and interaction of the editing elements E. The editing element E may be two-dimensional ( 2D) images, three-dimensional (3D) models, movies or sound signals. Moreover, the augmented reality editing module 60 editing the interactive content of the augmented reality application may include: setting, adjusting, or combining the position, order, size, angle, or control method of the editing element E. Alternatively, the editing element E may provide a virtual keyboard (not shown) for the user A to input information through the virtual keyboard.

In step S7 of FIG. 2, the augmented reality display module 70 shown in FIG. 1 displays the interactive content of the augmented reality application edited by the user A with the editing element E in a superimposed manner. On the image.

In step S8 in FIG. 2, the interactive content of the edited augmented reality application is recorded or stored in a file manner by the information storage module 80 in FIG. 1.

3A to 3C are schematic diagrams of an embodiment of the augmented reality application generation system and method of the present invention.

As shown in FIG. 3A, a user A wearing a smart glasses device B is in a space, and there is a table C in the space, and a specific Icon D (ie mark). After user A sees the icon D (marker), the image recognition tracking module and spatial positioning module of the augmented reality application generation system can identify and track the icon D to obtain three-dimensional spatial positioning information. User A can start editing the interactive content of the augmented reality application through the augmented reality editing module.

As shown in Figure 3B, the augmented reality editing module of the augmented reality application generation system can provide or display multiple virtual editing elements E (such as toy balls, teddy bears, cars, etc.) for user A to edit or use. At the same time, the user A can control through various gestures (such as gesture A1) or voice A2 to select the desired editing element E (such as teddy bear, car), and place the editing element E at the desired position (No. 3C As shown).

In addition, user A may not need any specific objects within the scope of the user A, and dynamically identify and locate the current image in three dimensions through, for example, Simultaneous Localization And Mapping (SLAM).

4A to 4C are schematic diagrams of another embodiment of the augmented reality application generating system and method of the present invention.

As shown in FIG. 4A, a user A wearing a smart glasses device B is in a space with a table C in the space, and a specific object F (such as a specific vase object, etc.) is placed on the table C, that is, an image , Markers, or their image feature points). When user A's area contains object F (image, mark or image feature point information), the image recognition and tracking module and spatial positioning module of the augmented reality application generation system can perform object F Identification and tracking to obtain three-dimensional spatial positioning information, user A can edit through augmented reality The editing module starts editing interactive content of augmented reality applications.

As shown in FIG. 4B, the augmented reality editing module of the augmented reality application generation system can provide or display multiple virtual editing elements E (such as virtual flowers) for user A to edit or use. At the same time, the user A can control through various gestures (such as gesture A1) or voice A2 to select the desired editing element E (such as the virtual flower of the first style), and place the editing element E at the desired position ( (Figure 4C).

In addition, in the embodiments of FIGS. 3A to 3C or the embodiments of FIGS. 4A to 4C, the user A can change the position arbitrarily in this space, but the position and angle of the user A are different. , The position and angle of the augmented reality image seen by the natural user A also changes.

User A can also interact with the augmented reality application generation system through human-machine interaction methods such as gesture A1 or voice A2 control. For example, the augmented reality editing module of the augmented reality application generation system is notified by voice to display more editing elements E, and the recognition of the hand touching one of the editing elements E in the augmented reality screen to know the editing Element E is clicked, and the selected editing element E is enlarged by recognizing the action of two hands stretching outwards, and the virtual keyboard is displayed by voice A2 notification. The gesture A1 touches the virtual keyboard or is controlled by voice A2. Know the contents of the input buttons. All of the above interaction methods can be used in an immersive environment without any physical contact.

It can be known from the above that in the augmented reality application generation system and method of the present invention, the augmented reality technology or the smart glasses device is further combined, and the user's current image signal (environment image) is captured and identified. ) Or external signals, such as audio signals, Dynamic editing of augmented reality applications in an immersive way to make editing of augmented reality applications more intuitive and convenient. In addition, users can dynamically edit user-specific or customized augmented reality applications and interactions through simple, intuitive or fast human-machine interaction control methods, such as gesture (gesture) or voice control, to facilitate the entire editing and Operation process. At the same time, the results of the editing and operation can be recorded or stored in the form of a file, and an augmented reality application program is provided to load the file, so as to achieve the desired way of augmented reality application when the user edits.

The above-mentioned embodiments merely exemplify the principles, features, and effects of the present invention, and are not intended to limit the implementable scope of the present invention. Anyone who is familiar with this technology can perform the above operations without departing from the spirit and scope of the present invention. Modifications and changes to the implementation form. Any equivalent changes and modifications made by using the disclosure of the present invention should still be covered by the scope of patent application. Therefore, the scope of protection of the rights of the present invention should be as listed in the scope of patent application.

Claims (20)

An augmented reality application generation system includes: a signal capture module that captures at least one image signal; and an image recognition tracking module that recognizes and tracks the image captured by the signal capture module The image or mark of the signal to obtain the image or mark position information of the image signal; a gesture recognition tracking module that recognizes and tracks the user's pose in the image signal captured by the signal acquisition module, and Obtain the user's attitude type information or attitude position information; a human-machine interactive analysis module that analyzes the image signal captured by the signal acquisition module and the user's attitude obtained by the gesture recognition tracking module Category information or attitude position information to generate corresponding human-machine interaction information; an augmented reality editing module that provides at least one editing element based on the human-machine interaction information of the human-machine interaction analysis module for the human-machine interaction information The user uses the editing element to edit interactive content of an augmented reality application; and an augmented reality display module that superimposes the augmentation edited by the user with the editing element Interactive applications throughout the content displayed on reality image. The system according to item 1 of the scope of patent application, wherein the signal acquisition module or the augmented reality display module is connected or integrated with the smart glasses device worn by the user. The system described in item 1 of the patent application scope, wherein the signal is captured The fetching module has a video capture unit and an audio capture unit, the video capture unit captures the image signal and its depth information, and the audio capture unit captures the sound signal. The system according to item 1 of the scope of patent application, wherein the image recognition and tracking module recognizes and tracks the feature points of the image or the mark through one of the following algorithms: scale-invariant feature conversion ( SIFT) algorithm, accelerated robust feature (SURF) algorithm, accelerated segmented feature test (FAST) algorithm, binary robust independent basic feature (BRIEF) algorithm, or directional BRIEF (ORB) algorithm. The system according to item 1 of the scope of patent application, wherein the augmented reality editing module provides the user to select different editing elements through multiple human-computer interactions to edit the visual presentation and interaction of the editing elements. , And the editing element is a two-dimensional (2D) image, a three-dimensional (3D) model, a movie, or a sound signal. The system described in item 1 of the scope of patent application, wherein the augmented reality editing module editing the interactive content of the augmented reality application includes setting the position, order, size, angle or control method of the editing element , Adjust, or combine. The system according to item 1 of the scope of patent application, wherein the editing element provides a virtual keyboard for the user to input information through the virtual keyboard. The system described in item 1 of the scope of patent application further includes a spatial positioning module, which generates a corresponding three-dimensional spatial determination based on the image or mark position information of the image signal, or the user's attitude and position information. Bits of information. The system according to item 8 of the scope of patent application, wherein the three-dimensional spatial positioning information is represented or stored in a matrix data manner. The system described in item 1 of the scope of patent application further includes an information storage module that records or stores the edited interactive content of the augmented reality application in a file manner. An augmented reality application generation method includes: capturing at least one image signal; identifying and tracking an image or mark of the image signal to obtain image or mark position information of the image signal; identifying and tracking the image signal in the image signal The user ’s posture to obtain the user ’s posture type information or posture position information; analyze the image signal and the user ’s posture type information or posture position information to generate corresponding human-computer interaction information; based on the human-computer interaction The information provides at least one editing element for the user to use the editing element to edit interactive content of the augmented reality application; and overlays the user with the augmented reality application edited by the editing element. Interactive content is displayed on the real-world image. The method described in item 11 of the scope of patent application is applied to the smart glasses device worn by the user. The method described in item 11 of the scope of patent application further includes capturing depth information of the image signal or sound signal. The method described in item 11 of the patent application scope further includes the following One of the algorithms identifies and tracks the feature points of the image or the image: scale-invariant feature conversion (SIFT) algorithm, accelerated robust feature (SURF) algorithm, accelerated segmented feature test (FAST) algorithm Method, binary robust independent foundation feature (BRIEF) algorithm, or directional BRIEF (ORB) algorithm. The method described in item 11 of the scope of patent application, further includes providing the user to select different editing elements through multiple human-machine interactions to edit the visual presentation and interaction of the editing elements, and the editing elements are two-dimensional (2D) images, three-dimensional (3D) models, movies or sound signals. The method according to item 11 of the scope of patent application, wherein editing the interactive content of the augmented reality application includes setting, adjusting, or combining the position, order, size, angle, or control method of the editing element. The method according to item 11 of the scope of patent application, wherein the editing element provides a virtual keyboard for the user to input information through the virtual keyboard. The method described in item 11 of the scope of the patent application further includes generating corresponding three-dimensional spatial positioning information based on the image or mark position information of the image signal, or the posture position information of the user. The method according to item 18 of the scope of patent application, wherein the three-dimensional spatial positioning information is represented or stored in a matrix data manner. The method described in item 11 of the scope of patent application further includes recording or storing the edited interactive content of the augmented reality application in a file manner.