KR102389762B1 - Space Formation and Recognition System through Digital Twin-linked Augmented Reality Camera and the method thereof - Google Patents

Abstract

The present invention relates to a space formation and recognition system through a digital twin-linked augmented reality camera and a method thereof and, more particularly, to a space formation and recognition system through a digital twin-linked augmented reality camera that can infer which part of a building a user has photographed by comparing a direction vector of the camera extracted from a 6DoF value through DT with a building that is hit by raycasting the direction vector on a corresponding point on a DT space, and a method thereof.

Description

Space Formation and Recognition System through Digital Twin-linked Augmented Reality Camera and the method thereof

The present invention relates to a space formation and recognition system and a method through a digital twin-linked augmented reality camera, and more particularly, by projecting a direction vector of a camera extracted from a 6DoF value through DT to a corresponding point on the DT space (raycasting) It relates to a space formation and recognition system through a digital twin-linked augmented reality camera capable of inferring which part of which building a user has photographed, and a method therefor, compared to a detected building.

The 3D spatial map currently provided to general users provides the shape of a building in the form of creating a 3D model through photogrammetry and applying a texture using photo data obtained through aerial photography. However, it still requires a lot of human manual work.

However, this method cannot implement precise images of individual buildings due to the nature of taking and matching a large number of photos, and there is no alternative to the case where the subject is blocked by obstacles such as street trees (low resolution, limitations of aerial/satellite photos) ).

Currently, a manufacturing method using a drone is being used, but it does not fundamentally solve the problem.

The present invention has been made to solve the above problems, in which the photos taken by the user are uploaded to the DT together with the posture information, and the camera's direction vector extracted from the 6DoF value through the DT and the augmented reality recognized through the augmented reality function. It is possible to infer which part of which building the user has photographed by comparing the spatial information with a building that is detected (hit) by projecting (raycasting) to the corresponding point on the DT space.

In addition, according to the present invention, the DT can increase the visual quality of each building through a spatial projection method in which the user projects pictures of the same building taken from different angles on a building object in the DT space.

In order to solve the above problems, the present invention includes an augmented reality device or augmented reality device comprising a 6DoF camera or an augmented reality camera and configured as a control module for controlling the same; GPS, WiFi, geomagnetism, and gyroscope modules that can identify the user's current location and posture information mounted in the augmented reality device; 6DoF (Degrees of Freedom) information obtained by summing the result data of the modules and the augmented reality spatial information to identify which direction the user is currently looking at, and the photos taken by the user through the augmented reality device are uploaded to the DT along with the posture information and the augmented reality spatial information, The camera's direction vector extracted from the 6DoF value through It infers which part of which building the user has photographed.

The DT includes data on the building and terrain information, the augmented reality spatial information recognized through the information photographed by the user and the augmented reality is projected on the DT as it is, and the DT is the user's vector with the building on the DT By calculating the collision point, it is inferred which building the user has photographed.

The 6DoF includes location information x, y, z or longitude, latitude, altitude, and POSE information including Roll, Yaw, and Pitch, and the augmented reality spatial information includes 3D spatial information such as buildings around the camera. The photo information taken and uploaded through the camera is reflected in the 3D object after spatial matching on the DT and provided to other users.

Using the space and administrative information on the building included in the digital twin, the labeling data required for artificial intelligence learning is automatically provided.

A plurality of pictures of the same building taken from different angles are projected onto the building object in the DT space to increase the visual quality of each building by a certain value or more.

The present invention uses 6DoF information obtained by summing measurement data of GPS, WiFi, geomagnetism, and gyroscope modules that can identify the user's current location and posture information included in the augmented reality device, allowing the user to identifying which direction it is facing; The photos taken by the user using the augmented reality device are uploaded to the DT along with the posture information, and the camera's direction vector extracted from the 6DoF value through the DT is projected (raycasting) to the corresponding point in the DT space, and the surrounding buildings in the DT space detecting; Detecting one building by comparing it with a plurality of surrounding buildings detected in order to infer which building the user has photographed using the augmented reality device among the buildings detected on the DT space; Comparing the 3D spatial information of the real building recognized through the augmented reality function with the spatial information of one building detected through the above process, detecting which area in the building was photographed; includes.

The DT includes data on building and terrain information, information captured by the user and augmented reality spatial information is projected on the DT as it is, and the DT calculates the point at which the user's vector collides with the building on the DT. , inferring which building the user has photographed; includes.

automatically providing labeling data necessary for learning of artificial intelligence by using space and administrative information about the building included in the DT; and projecting the pictures of the same building taken from different angles by the user on the building object in the DT space to increase the visual quality of each building by a certain value or more.

According to the present invention made as described above, since the photo information taken and uploaded by the user with the 6DoF camera is reflected in the 3D object after spatial matching on the DT and provided to other users, it is possible to provide more realistic image information to visitors. This helps local geospatial information providers such as local merchants to easily decorate their space realistically, and this is a method that can provide higher quality spatial information compared to the aerial photography method.

In addition, in the present invention, the information photographed by the user is projected as it is on the DT as described above, and since the DT has data on the building and terrain information, the DT can calculate the point at which the user's vector collides with the building on the DT, and this Through this, it is possible to infer which building the user has photographed.

In addition, the present invention includes a building information identification function that can automatically find out which building was photographed in the DT space using an image content analysis technique, and automatically determines which landmarks tourists take a lot of photos in in the tourism industry. to obtain a countable effect.

In addition, according to the present invention, since the digital twin already has the space and administrative information about the building present in the photo, the digital twin can automatically provide the labeling data required for artificial intelligence learning. This is a way to dramatically lower the learning cost of artificial intelligence that learns a space and utilize the digital twin space as a learning space for artificial intelligence.

1 is a view showing an inaccurate building image according to an embodiment of the present invention.
2 is a view showing an example of spatial projection according to an embodiment of the present invention.
3 is a view showing an example of image registration according to an embodiment of the present invention.
4 is a view showing an example of building information identification according to another embodiment of the present invention.
5 is a view showing a detailed configuration of a space formation and recognition system through a digital twin interlocking augmented reality camera according to the present invention.
6 is a diagram showing in detail the relationship between nodes and edges using a space formation and recognition system through a digital twin-linked augmented reality camera according to an embodiment of the present invention.
7 is a view showing a process of generating real-time data according to photo taking and coupon use using a space formation and recognition system through a digital twin interlocking augmented reality camera according to an embodiment of the present invention.
8 is a diagram illustrating a relationship between a plurality of nodes and edges created according to another embodiment of the present invention.
9 is a view showing (a) an existing map, (b) an example of internal logic, and the like in setting location information according to another embodiment of the present invention.
10 is a diagram showing an example of actual information-based DT projection according to another embodiment of the present invention.
11 is a view showing an example of an incident angle change according to a movement of a user's position according to another embodiment of the present invention.
12 is a diagram illustrating designation of a photographing area according to selection of four vertices of a photographing area according to another embodiment of the present invention.
13 is a view showing a display state of a request for designating a lower right vertex of a building according to another embodiment of the present invention.
14 is a diagram illustrating an augmented reality projection after spatial matching is completed according to another embodiment of the present invention and UV value designation for four vertices of a photographing area according to the process of FIG. 12 .
15 is a view showing an example of searching for a corresponding area in the DT building based on the augmented reality spatial information recognized through the augmented reality camera according to another embodiment of the present invention.
16 is a view showing an example of collecting various normal vectors from the complex appearance of a 3D mesh in a digital twin to create a representative plane and extracting one principal component vector from the generated plane according to another embodiment of the present invention.
17 is a diagram illustrating an example of a minimum bounding box that is a normal vector normalization result of FIG. 16 according to another embodiment of the present invention.
18 is a diagram illustrating a procedure for calculating the minimum bounding box of FIG. 16 according to another embodiment of the present invention.
19 is a diagram illustrating an example of Graham Scan, which is a convex Hull generation algorithm for calculating the minimum bounding box of FIG. 17 .
20 is a diagram illustrating an example of a convex hull calculated when the Graham Scan algorithm of FIG. 19 is completed.
FIG. 21 is a diagram illustrating an example of performing a Rotating Callipers algorithm using a convex hull, which is a process for calculating the minimum bounding box of FIG. 17 .
22 is a diagram illustrating an example of all bounding boxes calculated according to each trial of the Rotating Callipers of FIG. 21 .
23 is a diagram showing an example of matching a digital twin and the real world through rotation and position change according to another embodiment of the present invention.
24 is a view showing (a) an extracted building image and (b) an actual shape of each building surface according to another embodiment of the present invention.
25 is a diagram illustrating transformation of an original image into an actual form using projective transformation according to another embodiment of the present invention.
26 is a diagram illustrating an example of projection transformation using four vertices of each image according to another embodiment of the present invention.

In order to fully understand the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Embodiments of the present invention may be modified in various forms, and the scope of the present invention should not be construed as being limited to the embodiments described in detail below. This example is provided to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shape of elements in the drawings may be exaggerated to emphasize a clearer description. It should be noted that in each drawing, the same member is shown with the same reference numerals in some cases. In addition, detailed descriptions of well-known functions and configurations determined to unnecessarily obscure the gist of the present invention will be omitted.

Fig. 1 shows an inaccurate building image according to an embodiment of the present invention, Fig. 2 shows an example of spatial projection according to an embodiment of the present invention, and Fig. 3 is image registration according to an embodiment of the present invention Showing an example, Figure 4 is a view showing an example of building information identification according to another embodiment of the present invention.

As shown in Figures 2 to 5, the present invention includes an augmented reality device 100 including a 6DoF camera or an augmented reality camera and configured as a control module for controlling the same; A GPS module 201, a WiFi module 202, a geomagnetism module 203, and a gyroscope module 204 that can identify the user's current location and posture information in the control module 200 mounted in the augmented reality device. includes ;

6 Degrees of Freedom (6DoF: location information x, y, z or longitude, Latitude, Altitude + POSE information: Roll, Yaw, Pitch) information is used to identify which direction the user is currently looking at.

The augmented reality device may be replaced with other devices supporting augmented reality in addition to the augmented reality device.

In addition, by using 6DoF information obtained by summing up the data of the corresponding sensors of the augmented reality device, it is possible to identify from which point the user is currently looking in which direction.

The 6DoF includes x, y, z or longitude, latitude, and altitude information as location information, and Roll, Yaw, and Pitch information as POSE information.

And the photos taken by the user through the augmented reality device 100 are uploaded to DT (digital twin) together with posture information, and the direction vector of the camera extracted from the 6DoF value through DT is projected to the corresponding point in the DT space ( raycasting) is compared with a building to be detected (hit), and from among the detected buildings, which location of which building the user has photographed is inferred.

Therefore, according to the present invention, photo information taken and uploaded by a user with a 6DoF camera is reflected in a 3D object after spatial matching on the DT and provided to other users, so that more realistic image information can be provided to visitors.

According to an embodiment of the present invention, the DT includes data on the building and terrain information, the information captured by the user is projected on the DT as it is, and the DT is the user's posture vector collides with the building on the DT. By calculating the point, it is possible to infer which building the user has photographed.

To this end, the 6DoF includes location information x, y, z or POSE information including longitude, latitude, altitude and Roll, Yaw, and Pitch, and the photo information taken and uploaded through the camera is spatially aligned on the DT It is reflected in the 3D object and provided to other users.

Using the space and administrative information on the building included in the digital twin, the labeling data required for artificial intelligence learning is automatically provided.

Therefore, by projecting the pictures of the same building taken from different angles by the user on the building object in the DT space, the visual quality of each building can be increased by a certain value or more.

9 is a diagram illustrating (a) an existing map and (b) internal logic according to an embodiment of the present invention.

As shown in Figure 9, GPS has a relatively large error value depending on the user's surrounding environment, base station, and WiFi connection. If it is not correct, you can directly move the pin marker on the map to input your current location through the application (location information setting).

A marker exists in the center of the map displayed to the user through the application, and when the user moves the map screen and designates the marker on his or her current location, the application inputs the user's location at the coordinates closest to the marker.

In this case, it is expected that an error of several meters will occur in the process of converting the center coordinates of the screen input by the application into latitude and longitude coordinate values, but this can be resolved by correction of the AR SDK (Software Development Kit), which will be described later.

In this step, the position of the photographer is primarily arranged in the digital twin space, and finally spatially matched through the correction process using the AR SDK of FIGS. 15 to 23, which will be described later.

After the final spatial registration, the surrounding space (real world) viewed by the photographer using a smartphone can be tracked in the same way in the digital twin.

10 is a diagram illustrating an actual information-based DT projection for determining a subject building by interworking with DT.

Through the application, the direction the photographer is looking at from the location is converted into a vector, and the hit point is calculated by raycasting it in the digital twin space, so that the user can know which building he is currently looking at. .

Therefore, it is possible to display the image of the building selected in the digital twin by superimposing it on the screen of the augmented reality device by utilizing the augmented reality technology on the real building in the image that the user sees through the camera of the augmented reality device.

11 is a diagram illustrating an example of an incident angle change according to a movement of a user's position.

As shown in FIG. 11 , vector alignment in front of an area to be photographed (to secure an optimal photographing position) will be described in detail.

Now that the user has obtained information about which part of the building the user will take a picture of through the application's vertex display, the user needs to know the area of the photographed area.

In order to obtain the area, the vector viewed by the user is aligned with the normal vector (normal vector) of the building surface.

In this case, an image taken from the front of the subject as much as possible becomes an ideal texture image with minimal distortion.

To achieve this, the application compares and aligns the shooting direction vector viewed by the photographer's camera with the only normal vector coming from the surface of the subject building.

The user vector obtained from the camera by the application and the building surface vector (normal vector) obtained from DT are aligned on a straight line with the least distortion, and the size of the pier created by the two vectors in three-dimensional space. The photographer may try to confirm the shooting position with as little distortion as possible.

Since the dot product of two vectors is equal to the magnitude of the two vectors multiplied by the angle between the vectors (θ), the angle between the two vectors can be obtained by substituting the building normal vector and the user vector into the lower equation obtained by transforming the equation.

The angle is fed back to the user through the application, helping to secure the optimal camera shooting position.

The selection of four vertices of the imaging area will be described in detail as shown in FIG. 12 .

After the user designates the approximate center of the area to be photographed through the application, AR SDK can obtain the distance from the center point to each point when the user specifies the remaining four vertices of the area.

Alternatively, the same effect can be achieved by a method in which the photographer sequentially designates the center (focus) of the camera and the four vertices of the photographing area through the application.

The length of the line indicated by the dotted line and the angle indicated in green can be known through the application, and the user knows that the plane and the user are perpendicular through vector alignment, so using the trigonometric function or the Pythagorean principle, the length of the other two sides is also can be obtained, and the formula is:

In this case, D2 may be measured from the sensor of the augmented reality device linked to the application.

In designating the building vertex (optional) as shown in FIG. 13, the application displays any point on the left and right ends of the building to the user in order to know where the area of the building that the user wants to upload is located in the building. instruct

In FIG. 13 , the lower right vertex of the building was designated as an option.

If the application applies the above formula and data to the additionally designated point, how far the corresponding area is from the additional point can be equally calculated.

Using this, it is possible to calculate the area to be occupied by the photographed image in the area of the entire building.

As shown in FIG. 14 , the calculation of the UV value after selecting the four vertices of the imaging area will be described in detail.

When the user's location is completely matched to the space of the digital twin through the process of FIGS. 15 and 23, which will be described later, the user's real space and the space of the digital twin are in the same state, so the building of the digital twin is displayed in the form of augmented reality on the user's screen is displayed in combination with

Therefore, since the building of the digital twin is projected in the form of augmented reality on the real building viewed by the user through the camera, the area of the real building that the user wants to shoot can be arranged as the area of the building within the digital twin.

As shown in Fig. 15, a detailed look at the corresponding area search procedure in the digital twin for the augmented reality space.

Through the process of FIGS. 12 and 13, the position of the shooting area recognized by the user through the augmented reality space and the relative position between the user and the real building can be calculated, but the relative position between the user and the building area through the process of FIG. 15 In addition, the augmented reality space and the space on the digital twin can be finally matched.

Through the process of FIG. 16, which will be described later, a representative component of the building surface is extracted from the normal vectors of the building existing as a polyhedron in the digital twin space, and based on this, the normal vector and digital You can easily match the user's rotation values by comparing the normal vectors of the buildings present in the tween.

Since the digital twin and the user's rotation are matched, the normal vector of the real building wall detected by the user will coincide with the normal vector of the corresponding building face of the digital twin. If the area is searched, the user can specify the area within the building to be photographed.

Since the area of the building detected by the user and the area of the building in the digital twin space corresponding to it were obtained through the corresponding area search process, the user's location is identified as a digital twin by matching the piers and distances that appear as errors between the two areas through correction. It can be perfectly matched in space, an example of which is shown in FIG. 23 .

16 is a diagram for generalizing normal vectors of buildings of digital twins that are composed of polyhedra and have various directions to one normal vector on one side for smooth rotation correction between 3D spatial information of real buildings recognized by augmented reality and digital twins. This is a diagram showing an example, and ideally, a case of generalizing a normal vector is to generate a minimum bounding box like the example on the right of FIG. 17 .

In other words, unlike the building of a digital twin, which is composed of a polyhedron and has various normal vectors, which has a relatively large comparison error rate, the minimum bounding box has only four normal vectors, so it is easier to compare the rotation value with the normal vector of the wall of the building recognized through augmented reality. can be compared.

18 is a diagram illustrating a series of processes performed to obtain the minimum bounding box of an object as in the example of FIG. 17 . Since the digital twin has information on all vertices of 3D buildings existing in the digital twin space, information on vertices constituting an orthographic image generated when a target building is projected in the Z-axis direction can also be calculated.

A convex hull can be obtained by performing Graham Scan on a set of vertices constituting a 2D orthographic image in the Z-axis direction for a 3D building, and the process is shown in FIG. 19 .

The Graham Scan algorithm of FIG. 19 is a clockwise (CW) or counterclockwise direction in which a vector for a new point (next index) is specified based on a vector connecting two points sequentially specified in an array arranged in ascending order according to an angle based on one pole. As a method of generating a convex hull by connecting dots corresponding to the clockwise direction (CCW), CCW can be used as the vector direction detection (clockwise/counterclockwise) algorithm, and the formula of the CCW algorithm is as Equation 2.

20 is a diagram illustrating an example of a completed Convex Hull.

21 is an example showing a procedure for performing Rotating Callipers for the calculated convex hull. Since it was proved that the Minimum Bounding Box is in contact with at least one corner of the Convex Hull by Freeman and Shapira's theorem published in 1975, the Minimum Bounding Box is searched for the smallest area after performing Rotating Callipers on all corners. Bounding Box can be calculated.

22 is a diagram illustrating bounding boxes calculated for every corner according to the implementation of the algorithm described in FIG. 21 .

23 is a view showing a comprehensive rotation and position matching procedure through the position difference between the rotation matching process described in FIGS. 19 to 22 and the corresponding area in the building described in FIG. 15 .

As shown in FIG. 24 , when the texture image is corrected and reflected in the DT, the 3D spatial information in the screen obtained from the AR SDK calculated through the process shown in FIG. 12 and the location of the building in the specific screen by the user through the AR function If you use it, you can extract only the part corresponding to the building from the photograph.

Comparing the (left) extracted building image of FIG. 24 and (right) the actual image of each building surface, the image in which the user has reduced the error before and after shooting can be more precisely mapped to the DT mesh, so that the shooting result is at the shooting angle. It can be seen that the shape can be expressed in the same proportion and size as the actual shape, not in a distorted shape.

Through the application, the shape of a specific surface of the 3D mesh when viewed from a direction perpendicular to the surface (forward direction) can be known, so the original photographed image is converted into a forward shape.

The original image is converted into an actual form by using the transformation model projective transformation shown in FIG. 25 .

Since the application has information on the four vertices of the original image and the four vertices of the actual image, a model (3x3 matrix) representing the transformation relationship between the two images as shown in Equation 6 below can be obtained.

The matrix operation used for the projective transformation in the above application is as follows, and the pixel coordinates (x, y) are calculated after correcting the values corresponding to a, b, and c of the 3x3 matrix according to rotation, size transformation, movement, or projection. Multiply to get the transform coordinates (x', y').

26 is a diagram showing an example of projection transformation using four vertices of each image using the above-described methods.

Hereinafter, a space formation and recognition method through a digital twin interlocking augmented reality camera according to an embodiment of the present invention will be described in detail.

First, using the 6 DoF information obtained by summing the result data of the GPS, WiFi, geomagnetism, and gyroscope modules that can identify the user's current location and attitude information, the user is identified from which point and which direction the user is currently looking do.

In addition, by using the augmented reality function of the augmented reality device, spatial information of the building and surrounding areas that the user actually wants to photograph is detected.

The photos taken by the user are uploaded to DT along with posture information and augmented reality spatial information, and the camera's direction vector extracted from the 6DoF value through DT is projected (raycasting) to the corresponding point in the DT space to detect (hit) buildings and Compare.

In the comparing step, it is inferred which part of which building the user has photographed among the buildings detected using augmented reality spatial information.

The DT includes data on the building and terrain information, the information captured by the user is projected on the DT as it is, and the DT calculates the point where the user's vector collides with the building on the DT, so that the user Guess whether it was filmed.

By using the space and administrative information about the building included in the DT, it automatically provides the labeling data necessary for the learning of artificial intelligence.

By projecting photos of the same building taken from different angles by the user on the building object in the DT space, the visual quality of each building is raised by a certain value or more.

100: augmented reality device
200: control module
201 : GPS module
202: WiFi module
203: geomagnetism module
204: gyroscope module

Claims (8)

An augmented reality device comprising a 6DoF camera or an augmented reality camera and configured with a control module for controlling the same;
A GPS, WiFi, geomagnetic, gyroscope module capable of identifying the user's current location and posture information mounted in the augmented reality device; includes,
Using 6DoF (Degrees of Freedom) information obtained by summing the result data of the modules, identify which direction the user is currently looking at from which point,
The photos taken by the user through the augmented reality device are uploaded to the DT together with the posture information, and the direction vector of the camera extracted from the 6DoF value through the DT is projected (raycasting) to the corresponding point in the DT space and the building that is detected (hit) compare,
Inferring which part of which building the user has photographed among the detected buildings,
The DT includes data on the building and terrain information, the information captured by the user is projected on the DT as it is, and the DT calculates the point at which the user's posture vector collides with the building on the DT, so that the user A space formation and recognition system through a digital twin-linked augmented reality camera, characterized in that it infers whether a building was photographed.
delete According to claim 1,
The 6DoF includes location information x, y, z or longitude, latitude, altitude, and POSE information including Roll, Yaw, and Pitch,
The photo information taken and uploaded through the camera is reflected in the 3D object after spatial registration on the DT and provided to other users. Space formation and recognition system through a digital twin interlocking augmented reality camera.
According to claim 1,
Space formation and recognition system through a digital twin-linked augmented reality camera, characterized in that automatically providing labeling data required for learning of artificial intelligence by using space and administrative information about the building included in the digital twin.
According to claim 1,
Space formation and recognition through a digital twin-linked augmented reality camera, characterized in that the visual quality of each building in the DT space is raised by a certain value or more by projecting a plurality of photos of the same building taken from different angles on the building object in the DT space system.
Using 6DoF information obtained by summing the result data of GPS, WiFi, geomagnetism, and gyroscope modules that can identify the user's current location and posture information included in the augmented reality device, the user can determine which direction from which point. identifying whether you are looking;
The photos taken by the user using the augmented reality device are uploaded to the DT along with the posture information, and the camera's direction vector extracted from the 6DoF value through the DT is projected (raycasting) to the corresponding point in the DT space, and the surrounding buildings in the DT space detecting;
Detecting one building by comparing it with a plurality of surrounding buildings detected in order to infer which part of which building the user has photographed using the augmented reality device among the buildings detected on the DT space;
The DT includes data on the building and terrain information, the information captured by the user is projected on the DT as it is, and the DT calculates the point at which the user's posture vector collides with the building on the DT, so that the user A method of forming and recognizing a space through a digital twin-linked augmented reality camera comprising; inferring whether the building was photographed.
delete 7. The method of claim 6,
automatically providing labeling data necessary for learning of artificial intelligence by using space and administrative information about the building included in the DT;
The step of raising the visual quality of each building by more than a certain value by projecting the pictures of the same building taken from different angles by the user on the building objects in the DT space; space formation and recognition method through digital twin interlocking augmented reality cameras, including.

Images