TWI639960B - Augmented reality system and method thereof - Google Patents

Abstract

An augmented reality system, which can include an image capture module, an image recognition module, and an operation information storage module. The image capturing module can scan the target object to generate a three-dimensional image, and can capture a two-dimensional scene image containing the target object. The image recognition module may include an image database storing a plurality of three-dimensional models corresponding to the plurality of objects, and an image processing program may be executed on the three-dimensional images to generate image processing results, and compared with the three-dimensional models to select one of them. A three-dimensional model. The operation information storage module can store the operation information of the objects, and can select an operation information of the object corresponding to the selected three-dimensional model. The image capture module integrates the operational information of the selected object with the two-dimensional scene image to produce an integrated image.

Description

Augmented reality system and method thereof

The present invention relates to an augmented reality system, and more particularly to an augmented reality system integrating three-dimensional (3D) image recognition technology. The invention also relates to an image recognition method for the augmented reality system.

As IT integration applications such as Big Data, Industry 4.0, and Internet of Things have become the future trend, the role of Augmented Reality (AR) has become more and more important. In order to achieve the goal of automated production in the industrial chain, Integrating mobile devices, wireless communication, image recognition, data visualization and interface design, it can be seen that the development of augmented reality technology is market-oriented and a long-term industry that can be continuously developed.

Augmented reality is an application based on the identification of image features. The image captured by an image capture device (such as a video camera of a personal computer or a mobile device) is analyzed by an image recognition program to detect the corresponding image to be recognized. The feature point of the Marker or the target image, and calculate the position, rotation and zoom position of the image in space to project additional text, 2D or 3D information. The main purpose is to attach the computer information to the reality. In the world, users can see the expanded information of virtual and real integration. However, the augmented reality system of the prior art has many shortcomings to be improved.

For example, the user uses the mobile device to capture the two-dimensional (2D) image of the self-standard object for image recognition, and then the recognition result is corresponding to the relative information and is presented on the screen of the mobile device in two-dimensional, three-dimensional or text mode, and the core calculation is performed. Method for object recognition work for 2D image recognition, and 2D image recognition Knowledge technology has been in development for decades, with a high recognition rate (80% to 90%) in an ideal location. However, when applied in a practical environment, the influence of ambient light, shooting angle, shielding, background, etc., often severely interferes with the characteristics of the target image, resulting in a significant decrease in the recognition effect.

Therefore, how to propose an augmented reality system, which can effectively improve the recognition effect of the augmented reality system of the prior art has become an urgent problem.

In view of the above-described problems of the prior art, it is an object of the present invention to provide an augmented reality system to solve the problem of the poor performance of the augmented reality system of the prior art.

According to one of the objects of the present invention, an augmented reality system is provided, which may include an image capture module, an image recognition module, and an operation information storage module. The image capturing module can scan the target object to generate a three-dimensional image, and can capture a two-dimensional scene image containing the target object. The image recognition module may include an image database storing a plurality of three-dimensional models corresponding to the plurality of objects, and an image processing program may be executed on the three-dimensional images to generate image processing results, and compared with the three-dimensional models to select one of them. A three-dimensional model. The operational information storage module may include an operational information database storing operational information of the objects, and may select an operation information of the object corresponding to the selected three-dimensional model. The image capture module integrates the operational information of the selected object with the two-dimensional scene image to produce an integrated image.

In a preferred embodiment, the image capturing module can include a three-dimensional scanning unit and a two-dimensional image capturing unit; the three-dimensional scanning unit can scan the target object to generate a three-dimensional image, and the two-dimensional image capturing unit can extract the included A 2D scene image of the target object.

In a preferred embodiment, the image capture module can indicate the operation information of the selected object in the target object in the integrated image.

In a preferred embodiment, the image capturing module can further include a display unit, which can display the whole Image.

In a preferred embodiment, the image processing program can include a feature extraction program, a feature comparison program, and a feature retrieval program.

In a preferred embodiment, the image database can store a hash table corresponding to the three-dimensional model of each object.

In a preferred embodiment, the image database can be created by a feature extraction program and a feature compression program.

In a preferred embodiment, the image capture module can be a mobile device.

In a preferred embodiment, the three-dimensional scanning unit can be an infrared scanner.

According to one of the objects of the present invention, an image recognition method for augmented reality is further provided, which may include the steps of: providing an image database to store a plurality of three-dimensional models of a plurality of objects; and providing an operational information database for storing The operation information of the objects; scanning the 3D image of the target object; capturing the 2D scene image containing the target object; performing an image processing procedure on the 3D image to generate the image processing result; and the image processing result and the 3D model of the object Alignment to select one of the three-dimensional models; selecting an operational information of the object corresponding to the selected three-dimensional model; and integrating the operational information of the selected object with the two-dimensional scene image to produce an integrated image.

In a preferred embodiment, the image recognition method applied to the augmented reality may further include the following steps: displaying the integrated image.

In a preferred embodiment, the image recognition method applied to the augmented reality may further include the following steps: the target object in the integrated image indicates the operation information of the selected object.

In a preferred embodiment, the step of executing the image processing program on the three-dimensional image may include performing a feature extraction program, a feature comparison program, and a feature retrieval program on the three-dimensional image.

In a preferred embodiment, the step of providing the three-dimensional models of the objects may include: A hash table corresponding to the three-dimensional model of each object is stored.

In a preferred embodiment, the step of providing an image database may include: creating a image database through a feature extraction program and a feature compression program.

As described above, the augmented reality system and method thereof according to the present invention may have one or more of the following advantages:

(1) In an embodiment of the present invention, the augmented reality system can integrate the three-dimensional image recognition technology, so that the interference target is not affected by environmental light, out-of-focus, shooting angle, shielding, background, and the like. The characteristics of the image of the object can effectively enhance the recognition effect.

(2) In an embodiment of the present invention, the augmented reality system can indicate the operation information of the target object in the integrated image, so that the user can directly know the name of the target object and its operating state, and the use is even more Convenience.

1, 2‧‧‧ augmented reality system

11‧‧‧Image capture module

111‧‧‧2D image capture unit

112‧‧‧3D scanning unit

113‧‧‧Display unit

12‧‧‧Image Management Module

121‧‧‧Image database

13‧‧‧Operation Information Storage Module

131‧‧‧Operation Information Database

21‧‧‧ Tablet PC

22‧‧‧Image Management Server

23‧‧‧Operation Information Storage Server

I1‧‧‧2D scene image

I2‧‧‧3D image

IC‧‧‧ integrated image

S21~S28‧‧‧Step process

Figure 1 is a block diagram of a first embodiment of an augmented reality system of the present invention.

Figure 2 is a flow chart of the first embodiment of the present invention.

Figure 3 is a first schematic view of a second embodiment of the augmented reality system of the present invention.

Figure 4 is a second schematic view of a second embodiment of the augmented reality system of the present invention.

Figure 5 is a third schematic diagram of a second embodiment of the augmented reality system of the present invention.

Figure 6 is a fourth schematic view of a second embodiment of the augmented reality system of the present invention.

Figure 7 is a fifth schematic diagram of a second embodiment of the augmented reality system of the present invention.

The embodiments of the augmented reality system and method thereof according to the present invention will be described with reference to the related drawings, and the components in the drawings may be exaggerated in size and proportion for clarity and convenience of description. Presented in a reduced size. In the following description and/or claims, when a component is "connected" or "coupled" to another component, it can be directly connected or coupled to the other component or can be stored. In the case of an intervening component; when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements, and other words used to describe the relationship between the elements or layers should be interpreted in the same manner. For ease of understanding, the same elements in the following embodiments are denoted by the same reference numerals.

Please refer to FIG. 1, which is a block diagram of a first embodiment of the augmented reality system of the present invention. As shown in the figure, the augmented reality system 1 can include an image capture module 11, an image recognition module 12, and an operation information storage module 13.

The image capturing module 11 can include a two-dimensional image capturing unit 111, a three-dimensional scanning unit 112, and a display unit 113. The two-dimensional image capturing unit 111 can capture a two-dimensional image I1 including a target object; the three-dimensional scanning unit 112 can scan the target object to generate a three-dimensional image I2; in a preferred embodiment, the two-dimensional image capturing unit 111 It can be a camera or other similar device; the three-dimensional scanning unit 112 can be an infrared scanner or the like; the display unit 113 can be a screen or other similar device; and the image capturing module 11 can be a mobile phone or a notebook computer. , tablet or other similar device.

The image recognition module 12 can include an image database 121, which can store a plurality of three-dimensional models corresponding to a plurality of objects; the image recognition module 12 can execute an image processing program on the three-dimensional image I2 to generate image processing results, and Comparing with the three-dimensional models stored in the image database 121, and selecting a three-dimensional model corresponding to the image processing result according to the comparison result; in a preferred embodiment, the image processing module 12 can be Server or other similar device.

The operation information storage module 13 can include an operation information database 131 for storing operation information of the objects; the operation information storage module 13 can select an operation information of the object corresponding to the selected three-dimensional model; In an example, the operational information storage module 13 can be a server or other similar device.

Finally, the image capturing module 11 can display the operation information of the selected object and the two-dimensional scene image I1. Integration to generate an integrated image IC, and the integrated image IC can be displayed on the display unit 113; the integrated image IC can include the original two-dimensional scene, and the target object in the two-dimensional scene can indicate the operation information of the selected object. The operational information can include information such as the name of the target object and its operational status.

Therefore, the augmented reality system 1 can help the user to quickly familiarize with the facilities of a place, so it is suitable for various factories, power plants and the like to help the user to quickly enter the work condition to improve the work efficiency.

The augmented reality system 1 can scan the target object through the three-dimensional scanning unit 112 to generate the three-dimensional image I2, and then perform an image processing program for image recognition, so that the three-dimensional image recognition technology is integrated, so that the ambient light is not lost. The influence of factors such as focus, shooting angle, shelter, background, etc., severely interferes with the characteristics of the image of the target object, so that the recognition effect can be effectively improved, and the shortcomings of the conventional technique are indeed improved.

Please refer to FIG. 2, which is a flow chart of the first embodiment of the present invention. The image recognition method of the augmented reality system 1 of the embodiment may include the following steps: S21: providing an image database to store a plurality of three-dimensional models of the plurality of objects.

S22: Providing an operation information database to store operation information of the objects.

S23: Scan a three-dimensional image of the target object.

S24: Capture a 2D scene image containing the target object.

S25: Perform an image processing program on the 3D image to generate an image processing result.

S26: Align the image processing result with the three-dimensional model of the objects to select one of the three-dimensional models.

S27: Select an operation information of an object corresponding to the selected three-dimensional model.

S28: Integrate the operation information of the selected object with the two-dimensional scene image to generate an integrated image.

It is worth mentioning that the augmented reality system of the prior art is easy to be due to ambient light, shooting angle, The influence of factors such as the shelter and the background seriously interferes with the characteristics of the image of the target object, resulting in a significant decrease in the identification effect. In contrast, according to an embodiment of the present invention, the augmented reality system can integrate the three-dimensional image recognition technology, so that the target object is not seriously affected by ambient light, out-of-focus, shooting angle, shielding, background, and the like. The characteristics of the image can effectively enhance the recognition effect.

In addition, according to an embodiment of the present invention, the augmented reality system can indicate the operation information of the target object in the integrated image, so that the user can directly know the name of the target object and its operating state, which is more convenient to use. As can be seen from the above, the present invention has progressive patent requirements.

Please refer to FIG. 3, FIG. 4, FIG. 5, FIG. 6 and FIG. 7 , which are a first schematic diagram, a second schematic diagram and a third schematic diagram of a second embodiment of the augmented reality system of the present invention. The fourth schematic diagram and the fifth schematic diagram. As shown in FIG. 3, the augmented reality system 2 may include a tablet 21, an image recognition server 22, and an operation information storage server 23.

The tablet 21 can include a camera and an infrared scanner. The camera can capture a two-dimensional scene image I1 including a target object; the infrared scanner can scan the target object to generate a three-dimensional image I2; in this embodiment, the target object is a pressure tank of a power plant.

The image server 12 can include an image database that can store a three-dimensional model corresponding to all the machines and facilities of the power plant; the image server 22 can execute an image processing program on the three-dimensional image I2 to generate images. The result is processed and compared with the three-dimensional models stored in the image database, and then the three-dimensional model corresponding to the image processing result is selected according to the comparison result.

In the aspect of image database construction, it is necessary to first scan the target object with an infrared scanner, establish a three-dimensional point cloud data file, and process the 3D image feature database after processing the data through feature extraction program and feature compression program.

In terms of feature extraction technology, the present embodiment uses a point pair feature (PPF) to extract a point cloud data feature of a three-dimensional image, which describes the relative position and direction of two orientation points, as shown in FIG. 4, The two data points p ₁ and p ₂ have normal vectors n ₁ and n ₂ , respectively, where d = p ₂ - p ₁ , and a eigenvector F is obtained according to equation (1). Each eigenvector F contains 4 parameters. value, as in FIG. 4 in F1 ~ F4, F1 is a distance d between the coordinates of two points, F2 and d is the angle of the normal vector p ₁ n _1, F3 normal vector p d is an angle of ₂ n _2, and F4 The angle between n ₁ and n ₂ .

F ( p ₁ , p ₂ )=(∥ d ∥ ₂ , ∠ ( n ₁ , d ) , ∠ ( n ₂ , d ) , ∠ ( n ₁ , n ₂ )).......... ..............(1)

After the three-dimensional point cloud data is calculated by the point feature (PPF) randomly taking any two points in the model to calculate its feature vector F , each feature vector F is transformed by a hash function (Hash Function), and is compressed from 4 feature parameters into 1 An address (Index), which represents the location of the pair of feature vectors stored in the Hash Table. The subsequently generated feature vector is close to one of the existing clusters and is added to the cluster. If not, Create a new cluster. Each 3D image is generated by feature extraction and feature compression to generate respective hash tables. These hash tables are image data libraries for reference image groups of 3D image comparisons when performing image recognition operations.

The use of hash tables is faster in data retrieval than directly from eigenvalue calculations, and can achieve the effect of fast object comparison. This data distribution is called hashing, and the speed of accessing data can reach O(1). ), this is not possible with other data structures; the hash table construction process is shown in Figure 5.

In the augmented reality operation stage, the image processing program executed by the image recognition server 22 on the three-dimensional image I2 may include a feature extraction program, a feature comparison program, and a feature retrieval program. Image recognition server 22 can run on the three-dimensional point cloud infrared image 21 of the tablet, a scanner, after generating feature extraction procedure (PPF) data group of feature points, wherein a certain reference point cloud data points s _r corresponding to the image data When the point cloud data point m _r of a certain three-dimensional image in the library is different, the normal vector corresponding to the two point cloud data points s _r and m _r is the angle α, and the space coordinates from the scanned image and the database image can be obtained. Transform information (m _r , α). The complete coordinate transformation of the database image and the scanned image can be expressed by the following formula (2):

Therefore, the point cloud data pair ( m _r , m _i ) in the image database M ² can correspond to the point cloud data pair ( s _r , s _i ) in the scanned image S ² , this coordinate transformation is shown in Figure 6.

The steps of coordinate transformation are as follows: 1. First, the image database point cloud data point m _r uses the coordinate transformation T _{m → g} to the origin, and its normal vector Rotate to the x- axis; 2. The point cloud data point s _{r of the} scanned image is shifted to the origin by the coordinate transformation T _{s → g} , and its normal vector It also rotates to the x- axis; 3. Combine T _{m → g} , T _{s → g} and the rotation angle R ( α ) to obtain the coordinate transformation formula from the scanned image and the database image.

When performing the feature comparison program and the feature retrieval program, the optimal rotation coordinates are found by calculating the feature vectors of the scanned image and the database image, and the rotated coordinates are composed of 2D array accumulators and vote statistics, such as The accumulator space in Fig. 7, wherein the column is the number of all feature vectors of the 3D image of the database, and the number of samples of the rotation angle α of the column.

Before performing the voting operations, in addition to the image scanning 3D point cloud data points to s _r all point cloud data points s _i S is paired with s _r to generate a feature vector F _s ( s _r , s _i ), and the feature vector is output from the hash table in the image database to find the corresponding feature vector through the address output by the hash function ( m _r , m _i ), and calculate the coordinate system ( m _r , α) of the scanned image using equation (2), which corresponds to ( m _r , m _i ) and ( m _r , s _i ) The coordinate system ( m _r , α) is used as an index value (Index) to vote on the 2D array accumulator.

In the schematic diagram of the voting mechanism in Fig. 7, when the eigenvectors F _s of the point cloud data points s _r and s _i of the scanned 3D image are calculated, the hash table of the image database is compared and calculated to find a similarity. Pairing the complete model with the angle characteristics of the normal vector such as ( m _r , m _i ) and ( , ), and find the rotation angle α of the area coordinates, and then use the voting mechanism to vote for ( m _r , α), and return the maximum number of rotation angles as the similarity reference value.

In order to speed up the calculation of the voting mechanism, the coordinate transformation step can be disassembled into two parts, from the coordinate system of the database image to the intermediate coordinate system α _m , and from the space of the scanned image to the intermediate coordinate system α _s , as in the formula (3) shown

R _x ( α )= R _x (- α _s ) R _x ( α _m )............................... ........................(3)

Therefore, the formula (4) can be obtained:

Rx (‧) represents the transfer of a vector on the X-axis (as shown in Figure 6), and the transpose angles [ α , αm , αs ...] in parentheses. Rx is the space on the X axis after transposition, For the transposed Y-axis 0~ the space on the whole value. In equation (4), t is defined as the space of the x- axis and the positive y- axis, and is unique to the image database and any point cloud data points of the scanned image. Therefore, for each 3D image in the image database, the angle is α _s can be calculated and stored in advance from all feature vectors as an acceleration comparison calculation.

The above voting mechanism requires a lot of point cloud data to perform voting. However, the amount of point cloud data in 3D images is too large. The 2D array accumulator generates many similar or very different rotation angle data groups, so Pose Clustering technology can be used. Filter out unsuitable poses (Pose) to improve judgment accuracy. All calculated estimated posture data will be classified according to the similarity between its position and rotation angle, and the preset error value and rotation angle are smaller than They will be merged into the same class.

Through the above mechanism, the image recognition server 22 can generate image processing results for the three-dimensional image I2, and can compare with the three-dimensional models stored in the image database, and then select and correspond to the three-dimensional image I2 according to the comparison result. A three-dimensional model of the machine or facility.

The operation information storage server 23 may include an operation information database that stores operation information corresponding to all the machines and facilities of the power plant; the operation information storage server 23 may select an object corresponding to the selected three-dimensional model. Operational information.

Finally, the tablet 21 can integrate the operation information of the selected object with the two-dimensional scene image I1 to generate an integrated image IC, and display the integrated image IC on the screen of the tablet 21; the integrated image IC can include the original two The dimension scene, and the target object in the two-dimensional scene indicates the operation information of the selected object, and the operation information may include information such as the name of the target object and its operating state.

Therefore, in the present embodiment, the augmented reality system 2 can help the user to quickly familiarize with the various machines and facilities of a place (such as a power plant) to help the user quickly enter the work condition to enhance his work. effectiveness.

In summary, according to an embodiment of the present invention, the augmented reality system can integrate 3D image recognition technology, so that it is not subject to environmental light, out-of-focus, shooting angle, shielding, background, etc., and severe interference. The characteristics of the image of the target object can effectively enhance the recognition effect.

According to an embodiment of the present invention, the augmented reality system can indicate the operation information of the target object in the integrated image, so that the user can directly know the name of the target object and its operating state, which is more convenient to use.

It can be seen that the present invention has achieved the desired effect under the prior art, and is not familiar with the skill of the artist, and its progressiveness and practicability have been met with the patent application requirements.提出 Submit a patent application in accordance with the law, and ask your bureau to approve the application for this invention patent, in order to encourage creation, to the sense of virtue.

The above is intended to be illustrative only and not limiting. Any other equivalent modifications or alterations of the present invention are intended to be included in the scope of the appended claims.

Claims (13)

An augmented reality system includes: an image capture module that scans a target object to generate a three-dimensional image and captures a two-dimensional scene image including the target object; And storing an image data library corresponding to a plurality of three-dimensional models corresponding to the plurality of objects, and performing an image processing program on the three-dimensional image to generate an image processing result, and comparing with the three-dimensional models to select one of the three-dimensional images And a running information storage module, comprising: a running information database storing operation information of the objects, and selecting an operation information of the object corresponding to the selected three-dimensional model; wherein the image capturing mode The group integrates the selected operation information of the object with the two-dimensional scene image to generate an integrated image, the integrated image includes the two-dimensional scene image, and the target object in the two-dimensional scene image indicates the selected object Operational information. The augmented reality system of claim 1, wherein the image capturing module comprises a three-dimensional scanning unit and a two-dimensional image capturing unit; the three-dimensional scanning unit scans the target object to generate the three-dimensional image. And the two-dimensional image capturing unit captures the two-dimensional scene image including the target object. The augmented reality system of claim 1, wherein the image capturing module further comprises a display unit for displaying the integrated image. The augmented reality system of claim 1, wherein the image processing program comprises a feature extraction program, a feature comparison program, and a feature search program. The augmented reality system of claim 1, wherein the image database stores a hash table corresponding to the three-dimensional model of each of the objects. The augmented reality system of claim 1, wherein the image database is established by a feature extraction program and a feature compression program. The augmented reality system of claim 1, wherein the image capturing module is a mobile device. The augmented reality system of claim 2, wherein the three-dimensional scanning unit is an infrared scanner. An image recognition method for augmented reality comprises the steps of: providing an image database to store a plurality of three-dimensional models of a plurality of objects; providing an operational information database for storing operation information of the objects; scanning one a three-dimensional image of the target object; capturing a two-dimensional scene image including the target object; performing an image processing program on the three-dimensional image to generate an image processing result; comparing the image processing result with the three-dimensional model of the object Selecting one of the three-dimensional models; selecting an operation information of the object corresponding to the selected three-dimensional model; integrating the selected operation information of the object with the two-dimensional scene image to generate an integrated image, and integrating the image The two-dimensional scene image is included; and the target object in the integrated image indicates operation information of the selected object. The image recognition method applied to the augmented reality as described in claim 9 of the patent application further includes the following steps: displaying the integrated image. The image recognition method for augmented reality according to claim 9, wherein the step of executing the image processing program on the three-dimensional image comprises: performing a feature extraction program and a feature comparison program on the three-dimensional image. And a feature retrieval program. An image recognition method for augmented reality as described in claim 9 of the patent application scope, wherein The step of providing the three-dimensional models of the objects includes storing a hash table corresponding to the three-dimensional model of each of the objects. For the image recognition method applied to the augmented reality as described in claim 9, the step of providing the image database comprises: establishing the image database through a feature extraction program and a feature compression program.