KR102489927B1 - Method and Apparatus for entity tracking based on feature data independent of augmented reality engine - Google Patents

Abstract

It relates to a method and apparatus for tracking an entity based on feature data independent of an augmented reality engine, which generates a URI for feature data of an entity from a combination of a name of the entity and a name of an AR engine, and uses the URI to determine the characteristics of the entity. It is independent of the augmented reality engine as it is possible to track the entity in the augmented reality space by acquiring the feature data of the entity represented in a format that can be processed by the AR engine and executing the AR engine based on the acquired feature data. Entity tracking based on in-feature data becomes possible.

Description

Method and Apparatus for entity tracking based on feature data independent of augmented reality engine

It relates to a method and apparatus for tracking entities in an augmented reality space.

Augmented Reality (AR) is a technology that overlaps a virtual image on an image of a real space and displays it as a single image. In the augmented reality space corresponding to the real space, there is at least one entity to be displayed by overlapping the virtual image. do. Entity tracking technology is a representative technology provided by augmented reality applications because the movement of entities in the real space must be reflected in the augmented reality space. This entity tracking is performed by an augmented reality engine called and used by an augmented reality application. Hereinafter, the augmented reality engine will be simply referred to as an "AR engine".

Entity tracking technology can be classified into a 2D image tracking technology and a 3D object tracking technology. When the entity corresponds to a 2D image, the AR engine generates feature data representing characteristics of the 2D image from the 2D image and tracks the entity in the augmented reality space using the feature data. When the entity corresponds to a 3D object, the AR engine generates feature data representing characteristics of the 3D object from a plurality of images representing the 3D object and tracks the entity in the augmented reality space using the feature data. In this way, the algorithm for generating feature data is different depending on whether it is 2D image tracking or 3D object tracking.

Augmented reality applications are classified into native applications and web applications. A native application is an application developed based on a software development kit (SDK) provided by a platform such as Google Company's Android or iOS, and can be executed only on a platform supported by the SDK. A web application is an application written in a web language such as HTML (Hypertext Markup Language), CSS (Client Side Script), JS (Java Script), etc., and can be executed on both Android and iOS.

Since these various types of augmented reality applications use different AR engines, and the format of feature data used by each AR engine is different, each augmented reality application must create feature data in a format suitable for the AR engine it uses. . Accordingly, a conventional augmented reality application includes a feature data generation algorithm dependent on an augmented reality engine.

Because each AR engine has a different feature data generation algorithm and feature matching algorithm, some AR engines show superior tracking performance for 2D images or superior tracking performance for specific shapes of entities compared to other AR engines. It may show low tracking performance for other types and shapes of entities such as dimensional objects. Accordingly, development of augmented reality applications using various types of AR engines is being attempted, but augmented reality applications must include a plurality of feature data generation algorithms for all AR engines that are used, so development of augmented reality applications This becomes increasingly difficult, and there is a problem that the size of the augmented reality application increases.

It is independent of the augmented reality engine that can perform entity tracking in the augmented reality space by acquiring feature data independently of the augmented reality engine and executing the AR engine without using a feature data generation algorithm dependent on the augmented reality engine. An object of the present invention is to provide a feature data-based entity tracking method and apparatus. Another object of the present invention is to provide a computer-readable recording medium on which a program for executing the above-described entity tracking method is recorded on a computer. It is not limited to the technical problems as described above, and another technical problem may be derived from the following description.

According to an aspect of the present invention, a method for tracking entities based on feature data independent of an augmented reality engine includes a uniform resource identifier (URI) for feature data of a first entity from a combination of a name of a first entity and a name of a first AR engine. ) generating; acquiring feature data of the first entity representing the feature of the first entity in a format processable by the first AR engine by using a URI of the feature data of the first entity; and performing tracking of the first entity in an augmented reality space by executing the first AR engine based on the acquired feature data of the first entity.

The generating of the URI for feature data of the first entity may include generating a URI for feature data of the first entity from a combination of a host name of a resource server, a name of the first entity, and a name of the first AR engine. Generating and obtaining feature data of the first entity may include sending the first entity to a resource server corresponding to a host of the URI for feature data of the first entity using the URI for feature data of the first entity. Feature data of the first entity may be obtained from the resource server by requesting feature data of the first entity.

The entity tracking method may include generating a URI of the first entity from a combination of a host name of the resource server and a name of the first entity; and detecting a first AR engine to be used for tracking the first entity from among the plurality of types of AR engines, wherein generating a URI for feature data of the first entity A URI for feature data of the first entity may be generated by adding the name of the detected first AR engine to the URI of the first entity.

The entity tracking method further includes detecting an entity tracking method to be used by the first AR engine for the first entity, and generating a URI for feature data of the first entity. A URI for feature data of the first entity may be generated by adding the name of the detected first AR engine and the name of the detected entity tracking method to the URI of the first entity.

The detecting of the entity tracking method may include detecting an entity tracking method to be used by the first AR engine for the first entity according to whether the first entity corresponds to a 2D image or a 3D object. there is.

The entity tracking method further includes detecting at least one entity in an augmented reality space represented by the augmented reality content from augmented reality content, and generating a URI for feature data of the first entity comprises the detection A URI for feature data of the first entity may be generated from a combination of a name of the first entity among at least one entity and a name of the first AR engine.

The entity tracking method may further include generating a URI for feature data of the second entity from a combination of a name of the second entity and a name of a second AR engine; acquiring feature data of a second entity representing features of the second entity in a format processable by the second AR engine by using a URI of the feature data of the second entity; and performing tracking of the second entity in an augmented reality space by executing the second AR engine based on the acquired feature data of the second entity.

The entity tracking method may further include determining whether the first AR engine requires feature data or an original image of the first entity to track the first entity, and In the generating of the URI for data, if it is determined that the first AR engine requires the feature data of the first entity, a URI for the feature data of the first entity may be generated.

In the entity tracking method, when it is determined that the first AR engine requires the original image of the first entity, information about the original image of the first entity is obtained from a combination of the host name of the resource server and the name of the first entity. generating a URI; acquiring an original image of the first entity by using a URI of the original image of the first entity; and performing tracking of the first entity in the augmented reality space by executing the first AR engine based on the obtained original image of the first entity.

According to another aspect of the present invention, a computer-readable recording medium recording a program for executing the entity tracking method on a computer is provided.

According to another aspect of the present invention, an entity tracking device based on feature data independent of an augmented reality engine generates a URI for feature data of a first entity from a combination of a name of a first entity and a name of a first AR engine. processing unit; and obtaining feature data of the first entity in which a feature of the first entity is expressed in a format processable by the first AR engine using a URI of the feature data of the first entity, and the acquired feature of the first entity and an AR processing unit performing tracking of the first entity in an augmented reality space by executing the first AR engine based on data.

A URI for the feature data of the entity is generated from the combination of the name of the entity and the name of the AR engine, and the feature data of the entity in which the feature of the entity is expressed in a format that can be processed by the AR engine is obtained using the URI, and obtained in this manner. As it is possible to track an entity in an augmented reality space by executing an AR engine based on the acquired feature data, it is possible to track an entity based on feature data independent of the augmented reality engine.

In this way, the augmented reality application can perform tracking of the entity in the augmented reality space by obtaining the feature data of the entity using the URI independently of the augmented reality engine and executing the AR engine based on the feature data. eliminates the need to include feature data generation algorithms that are dependent on the augmented reality engine. That is, since the augmented reality application includes a feature data acquisition algorithm independent of the augmented reality engine, there is no need to include a feature data generation algorithm dependent on the augmented reality engine.

When an augmented reality application uses a plurality of AR engines, it is not necessary to include several algorithms for generating feature data from an original image of an entity for each AR engine, enabling the development of an efficient augmented reality application. Entity tracking in the augmented reality space according to the present invention can be achieved only by obtaining the feature data of the entity using the URI for the feature data of the entity, and calling and executing the AR engine based on the feature data. It is possible to provide a general-purpose augmented reality application that does not include an engine-dependent feature data generation algorithm.

As the augmented reality application does not need to include a feature data generating algorithm for each AR engine, the size of the augmented reality application can be reduced, and thus the amount of work required to develop the augmented reality application can be reduced. When developing an augmented reality application, it is very easy to develop an augmented reality application because most of the algorithms, such as the algorithm for obtaining feature data of an entity by using the URI of the entity's feature data, can be reused as they are in the existing augmented reality application. It happens.

1 is a configuration diagram of an AR client 1 according to an embodiment of the present invention.
2 is a diagram showing an example of feature data for a two-dimensional image.
3 is a diagram illustrating an example of feature data for a 3D object.
4 is a flowchart of an entity tracking method according to an embodiment of the present invention.
FIG. 5 is a diagram showing an example of a process of acquiring feature data of the AR client 1 shown in FIG. 1 .
6 is a diagram illustrating an example of execution of a plurality of augmented reality applications according to another embodiment of the present invention.
FIG. 7 is a diagram showing an example of a URI table stored in the resource server 2 shown in FIG. 1 .

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Embodiments of the present invention to be described below do not use a feature data generation algorithm dependent on the augmented reality engine, but acquire feature data independently of the augmented reality engine and execute the AR engine to track entities in the augmented reality space. A method and apparatus for tracking an entity based on feature data independent of an augmented reality engine that can be performed.

Hereinafter, the augmented reality engine-independent feature data-based entity tracking method may be briefly referred to as "entity tracking method", and the augmented reality engine-independent feature data-based entity tracking device may be briefly referred to as "entity tracking device". Or it may be called an "AR client". Hereinafter, the term "Augmented Reality" may be briefly referred to as "AR" and "URI" is an abbreviation of "Uniform Resource Identifier".

1 is a configuration diagram of an AR client 1 according to an embodiment of the present invention. Referring to FIG. 1, the AR client 1 according to this embodiment includes a main processor 11, an AR processing unit 12, a URI processing unit 13, a communication module 14, an input module 15, and an output module ( 16), a camera module 17, a sensor module 18, a storage 19, and a bus 20. The AR client 1 according to the present embodiment tracks at least one entity existing in the augmented reality space, and overlaps and displays a virtual image on the tracked entity. Therefore, the AR client 1 according to this embodiment may be called an entity tracking device.

The main processor 11 executes a certain application, and according to the program code of the application, the AR processing unit 12, the URI processing unit 13, the input module 15, the output module 16, the communication module 14, and the camera It controls the module 17 and the sensor module 18, and reads data from the storage 19 or stores data in the storage 19. In particular, the main processor 11 of this embodiment executes an augmented reality application described below. Examples of the main processor 11 include a central processing unit (CPU) and an application processor (AP).

The AR processing unit 12 acquires, interprets, and renders augmented reality content according to the execution of an augmented reality application. The URI processing unit 13 generates a URI for each entity, a URI for feature data of a certain entity, and a URI for an original image of a certain entity according to the execution of the augmented reality application. In this embodiment, an original image of an entity means data representing a 2D image or a 3D image of the entity. The AR processing unit 12 and the URI processing unit 13 are processors that operate according to the execution of the augmented reality application, and may be physically the same processor as the main processor 11, or the main processor 11, such as a graphics processing unit (GPU). It may also be a separate dedicated processor used by

The communication module 14 performs communication with the resource server 2 or other servers through a network such as the Internet. The communication module 14 may perform communication with the resource server 2 or other servers according to various communication methods such as Long Term Evolution (LTE) and Wi-Fi. The input module 15 receives a user's command, selection, information, and the like. Examples of the input module 15 include a touch panel, a keyboard, and a mouse. The output module 16 is composed of a combination of an image output device such as a display panel and an audio output device such as a speaker to provide audio-visual output to the user.

The camera module 17 creates an image of a real space by capturing a space where the AR client 1 is currently located. The sensor module 18 detects the current position and posture of the AR client 1 using a gyroscope, accelerometer, GPS (Global Positioning System) sensor, etc. The current location and attitude information of the client 1 are generated. The location information of the AR client 1 may be expressed as three-dimensional coordinate values of x, y, and z in the augmented reality space corresponding to the three-dimensional coordinate system. The posture information of the AR client 1 includes a roll value representing the rotation angle in the x-axis direction, a pitch value representing the rotation angle in the y-axis direction, and a yaw value representing the rotation angle in the z-axis direction. can be expressed as

The storage 19 stores augmented reality applications and data used in their execution process. According to the example shown in FIG. 1 , the storage 19 stores augmented reality application A, AR engine A, AR engine B, augmented reality content, and the like. Each AR engine is a function program that is called by an augmented reality application to perform entity tracking. The bus 20 plays a role of transferring data between the above-described various components such as the main processor 11, the AR processing unit 12, and the URI processing unit 13.

The resource server 2 has a database of feature data layered and stored according to a plurality of entities, a plurality of AR engines, and a plurality of entity tracking methods using a URI scheme, and when a certain URI is received, corresponding feature data is provided. do.

2 is a diagram showing an example of feature data for a two-dimensional image. 2(a) shows a 2D image of a certain house, and FIG. 2(b) shows feature data for the 2D image shown in FIG. 2(a). Cross dots indicated in (b) of FIG. 2 correspond to feature points of the 2D image shown in (a) of FIG. 2 . Feature data for a 2D image is binary data representing these feature points. (b) of FIG. 2 shows binary data visualized to help understand feature data of a 2D image. Each AR engine detects a part corresponding to the feature data of the 2D image in each of a plurality of real space images continuously input from the camera module 17, and changes in the detected part in the real space represented by the plurality of real space images. From the trend, each entity in the augmented reality space corresponding to the real space can be tracked.

3 is a diagram illustrating an example of feature data for a 3D object. In (a) of FIG. 3, any one of a plurality of two-dimensional images of a cat doll corresponding to a three-dimensional object is shown. In order to express a 3D object, a plurality of 2D images of a cat doll photographed from various angles are required. 3(b) shows feature data generated from a plurality of two-dimensional images of a cat doll. The points of the 3D coordinate system shown in (b) of FIG. 3 correspond to the feature points of the cat doll corresponding to the 3D object. Feature data for a 3D object is binary data representing these feature points. (b) of FIG. 3 shows binary data visualized to help understand feature data of a 3D object. Each AR engine detects a portion corresponding to the feature data of a 3D object in each of a plurality of real space images continuously input from the camera module 17, and changes in the detected portion in the real space represented by the plurality of real space images. From the trend, each entity in the augmented reality space corresponding to the real space can be tracked.

4 is a flowchart of an entity tracking method according to an embodiment of the present invention. Referring to FIG. 4 , the entity tracking method according to the present embodiment is composed of steps sequentially processed by the AR client 1 shown in FIG. 1 . Therefore, even if the content is omitted below, the content described above regarding the AR client 1 shown in FIG. 1 is also applied to the entity tracking method to be described below. The entity tracking method according to the present embodiment can be implemented as an augmented reality application, and is executed by the main processor 11, the AR processing unit 12, the URI processing unit 13, and the like of the AR client 1. Hereinafter, each component such as the main processor 11, the AR processing unit 12, and the URI processing unit 13 of the AR client 1 will be described in detail with reference to FIG. 4 .

In step 41, the main processor 11 of the AR client 1 executes the augmented reality application A when receiving a command to execute the augmented reality application A according to the present embodiment from the user through the input module 15. Augmented reality application A is downloaded from the resource server 2 or another server and installed in the AR client 1 before its execution. Augmented reality application A may be provided by the resource server 2 or may be provided by another server. The augmented reality application A may be written in a computer programming language such as Java. The following steps are performed by the AR processing unit 12 and the URI processing unit 13 according to the execution of the augmented reality application A.

In step 42, the AR processing unit 12 of the AR client 1 displays information on various types of augmented reality content that can be provided by the augmented reality application A to the user through the output module 16. The user selects any one augmented reality content that the user wants to experience among several types of augmented reality content indicated by the displayed information. When the AR processing unit 12 receives user selection information for any one of at least one augmented reality content that can be provided by the augmented reality application A through the input module 15, any one augmented reality content indicated by the user selection information is displayed. It is acquired from the resource server 2 or other servers through the communication module 14. The augmented reality content may be provided by the resource server 2 or other servers. The communication module 14 transmits a request for augmented reality content of the AR processing unit 12 to the resource server 2 or other server through the network, and receives the augmented reality content from the resource server 2 or other server to the AR processing unit ( 12) plays a role in transmitting.

In step 43, the AR processing unit 12 of the AR client 1 analyzes the augmented reality content received in step 44 to detect at least one entity in the augmented reality space represented by the augmented reality content from the augmented reality content. Augmented reality content is content that represents each scene of the augmented reality space in a structured format, in which at least one virtual image is overlapped with an image of the real space, and includes information on at least one entity existing in the augmented reality space, e.g. For example, it contains the name of each entity. In this way, the AR processing unit 12 may detect the name of each entity in the augmented reality space by interpreting the augmented reality content.

Each entity existing in the augmented reality space is an element in which a virtual image corresponding to each entity is overlapped and displayed, and may be classified as a 2D image or a 3D object. Each entity existing in the augmented reality space must be tracked because a virtual image is displayed at its current location. If an entity is a 3D object, its position and posture may also be tracked because a virtual image may be differently displayed according to its current posture. In this embodiment, tracking an entity means tracking the position of an entity or tracking the position and posture of an entity.

Such augmented reality content may be written in a markup language such as XML (eXtensible Markup Language). The augmented reality content of this embodiment may represent an image of a single frame or an image of multiple frames. The augmented reality content of this embodiment may be mixed reality content displayed as an augmented reality image or a virtual reality image according to a rendering mode.

In step 44, the URI processing unit 13 generates a Uniform Resource Identifier (URI) for each entity from a combination of the host name of the resource server 2 and the name of each entity detected in step 43. The URI processor 13 may generate a URI for each entity by adding the name of each entity detected in step 42 to the host name of the resource server 2. Taking augmented reality content for displaying by overlapping a virtual image representing the address of a house existing in a certain real space as an example, the URI processing unit 13 assigns an entity to the host name "server.com" of the resource server 2. By adding the name "house" of the entity, the URI "http://server.com/house" can be created.

Taking augmented reality content for overlapping and displaying a virtual image representing a facial expression change according to the weather of the real space on the face of a cat doll existing in a certain real space as an example, the URI processing unit 13 is the host of the resource server 2 The entity's URI "http://server.com/cat" can be created by adding the entity's name "cat" to the name "server.com". The host name "server.com" of the resource server 2 may be included in augmented reality applications and may also be included in augmented reality content.

In step 45, the AR processing unit 12 detects an AR engine to be used for tracking each entity detected in step 41 from among a plurality of types of AR engines. The AR processing unit 12 may detect the name of the AR engine to be used for tracking the entity from the setting information of the augmented reality application A or the result of analyzing the augmented reality content in step 43 . Augmented reality content may include the name of an AR engine that is most suitable for tracking each entity that exists in the augmented reality space it represents. Augmented reality application A may be designed so that a user can select any one of multiple types of AR engines. In this case, the user may select one of a plurality of types of AR engines in the process of setting the execution environment of the augmented reality application A.

In step 46, the AR processing unit 12 determines which of the feature data and original image of the entity for tracking the entity detected in step 43 by the AR engine detected in step 45 based on the attribute of the AR engine detected in step 45. check if you need it In step 46, if it is determined that the AR engine requires feature data of an entity, step 47 is performed. In step 46, if it is determined that the AR engine requires the original image of the entity, step 410 is performed. For example, if the properties of AR engine A are set to always perform the process of generating its feature data from a 2D image before tracking the 2D image, then AR engine A will use the 2D image data, not the feature data. need

In step 47, the AR processor 12 detects an entity tracking method to be used by the AR engine detected in step 45 for each entity detected in step 43. More specifically, the AR processing unit 12 determines whether each entity detected in step 43 corresponds to a 2D image or a 3D object, and the AR engine detected in step 45 for each entity detected in step 43. The entity tracking method to be used can be detected. The 2D image tracking method and the 3D object tracking method in the augmented reality space are different depending on the data format between the 2D image and the 3D object.

In step 48, the URI processing unit 13 adds the name of the AR engine detected in step 45 and the name of the entity tracking method detected in step 47 to the URI for each entity generated in step 44, so that each entity detected in step 43 Create URIs for feature data. In this way, the URI processing unit 13 determines the host name of the resource server 2, the name of each entity detected in step 43, the name of the AR engine detected in step 45, and the name of the entity tracking method detected in step 47. From the combination, a URI for feature data of each entity detected in step 43 is generated.

FIG. 5 is a diagram showing an example of a process of acquiring feature data of the AR client 1 shown in FIG. 1 . For example, the name of the entity detected in step 43 is the name "house" of the 2D image shown in FIG. 2, the name of the AR engine detected in step 45 is "AREngineA", and the entity tracking detected in step 47. If the name of the method is “ImageTracking”, the URI for feature data of the 2D image entity shown in FIG. 2 becomes “http://server.com/house/AREngineA/ImageTracking”. For example, the name of the entity detected in step 43 is the name "cat" of the 3D object shown in FIG. 3, the name of the AR engine detected in step 45 is "AREngineB", and the entity tracking detected in step 47. If the name of the method is “ObjectTracking”, the URI for the feature data of the 3D object entity shown in FIG. 3 becomes “http://server.com/house/AREngineB/ObjectTracking”.

The AR processing unit 12 and the URI processing unit 13 are processors that operate according to the execution of the augmented reality application and operate according to the program code of the augmented reality application, so the augmented reality application provides a URI and feature data to help understanding of the present embodiment. It is shown in the form of obtaining.

In step 49, the AR processing unit 12 obtains feature data of each entity in which the characteristics of each entity are expressed in a format processable by the AR engine detected in step 45 using the URI of the feature data of each entity generated in step 49. do. The AR processing unit 12 accesses the resource server 2 corresponding to the host of the URI using the URI for the feature data for each entity generated in step 49, and retrieves the feature data of each entity representing the characteristics of each entity. request, and feature data of each entity is obtained from the resource server 2 as a response to the request. The communication module 14 transmits a request of the AR processing unit 12 to the resource server 2 through the network, receives a response to the request from the resource server 2, and transmits it to the AR processing unit 12. do

As shown in FIG. 5, if the URI for the feature data of the 2D image entity shown in FIG. 2 is "http://server.com/house/AREngineA/ImageTracking", the AR client 1 uses this URI. By accessing the resource server 2, the feature data of the 2D image entity shown in FIG. 2 may be obtained from the resource server 2. As shown in FIG. 5, if the URI for the feature data of the 3D object entity shown in FIG. 3 is "http://server.com/house/AREngineB/ObjectTracking", the AR client 1 uses this URI. By accessing the resource server 2, the feature data of the 3D object entity shown in FIG. 3 may be acquired from the resource server 2.

In step 410, the URI processing unit 13 adds “originalImage” to the URI for each entity generated in step 44, thereby generating a URI for an original image of each entity detected in step 43. Here, the original image of each entity means data representing a 2D image or a 3D image of each entity. Since the feature data of each entity has a very small data size compared to the original image of each entity, entity tracking can be performed efficiently.

In step 411, the AR processor 12 acquires the original image of each entity detected in step 43 using the URI of the original image of each entity generated in step 49. The AR processing unit 12 accesses the resource server 2 using the URI for the original image for each entity generated in step 49, requests the original image of each entity, and as a response to the request, the resource server 2 ) to obtain the original image of each entity. The communication module 14 transmits a request of the AR processing unit 12 to the resource server 2 through the network, receives a response to the request from the resource server 2, and transmits it to the AR processing unit 12. do Here, the original image of each entity means data representing a two-dimensional image of each entity.

In step 412, the AR processing unit 12 executes the AR engine detected in step 45 based on the feature data obtained in step 49 or the original image obtained in step 411 to augment the augmented reality space represented by the augmented reality content received in step 42. performs tracking of each entity in When the AR engine detected in step 45 is executed based on the original image obtained in step 411, the AR engine generates feature data of the original image from the original image. The augmented reality space means a space to be expressed by augmenting the augmented reality content using a virtual image in the real space where the AR client 1 is currently located, and may be all or part of the real space.

The AR processing unit 12 detects a portion corresponding to feature data of each entity in each of a plurality of real space images continuously input from the camera module 17, and detects a plurality of real space images continuously input from the camera module 17. The position of each entity in the augmented reality space can be tracked from the change in position of the detection part in the real space represented by the image. The AR processing unit 12 may also track the posture of each entity in the augmented reality space from the posture change trend of the detection part in the real space represented by a plurality of images continuously input from the camera module 17 .

Subsequently, the AR processing unit 12 renders the augmented reality content received in step 42, and places each entity at the position of each entity tracked in step 412 for each of a plurality of real space images input in real time from the camera module 17. Create an augmented reality image that overlaps the virtual image. The AR processing unit 12 may refer to the position and posture of each entity tracked in step 412 for each of the plurality of real space images, and rotate the virtual image of each entity according to the change in the posture of each entity so as to overlap the virtual images. The output module 16 displays the augmented reality image thus generated. Since the embodiment shown in FIG. 4 relates to an entity tracking method, this step is omitted in FIG. 4 .

6 is a diagram illustrating an example of execution of a plurality of augmented reality applications according to another embodiment of the present invention. According to the example shown in FIG. 6 , a plurality of augmented reality applications and a plurality of types of AR engines correspond to each other on a one-to-one basis. In another example, one augmented reality application may use multiple types of AR engines, and in another example, multiple augmented reality applications may use one type of AR engine. A plurality of augmented reality applications may be installed in one AR client 1, and each may be installed in each of the plurality of AR clients 1.

When the augmented reality application A is executed and the detected entity is "entity A", and the properties of AR engine A are set to be executed based on the feature data of entity A, steps 41 to 49 and 412 are performed and the entity A tracking is done. That is, the URI processing unit 13 includes the host name of the resource server 2, the name of entity A among at least one entity detected in step 43, the name of AR engine A detected in step 45, and the entity detected in step 47. A URI for the feature data of entity A is generated from the combination of names of tracking methods. Next, the AR processing unit 12 obtains feature data representing the characteristics of entity A in a format processable by AR engine A using this URI.

Next, the AR processing unit 12 executes the AR engine A based on the feature data obtained in this way to track the entity A in the augmented reality space. The AR processing unit 12 and the URI processing unit 13 are processors that operate according to the execution of the augmented reality application and operate according to the program code of the augmented reality application, so the augmented reality application provides a URI and feature data to help understanding of the present embodiment. It is shown in the form of obtaining.

As described above, each AR engine is a kind of function program that performs entity tracking independently of an augmented reality application, and the AR processing unit 12 calls the AR engine A, inputs feature data of the entity A, and converts the AR engine A to the AR engine A. run Subsequently, the AR engine A returns to the AR processing unit 12 the result of tracking the entity by the AR engine A, for example, the location of the entity A, to the AR processing unit 12. Next, the AR processing unit 12 overlaps the virtual image of each entity A at the position of entity A with respect to each of a plurality of real space images input from the camera module 17 in real time, thereby generating an augmented reality image for each real space image. generate

When the augmented reality application B is executed and the detected entity is "entity B", and the properties of AR engine B are set to be executed based on the feature data of entity B, steps 41 to 49 and 412 are performed and the entity B tracking is done. That is, the URI processing unit 13 includes the host name of the resource server 2, the name of entity B among at least one entity detected in step 43, the name of AR engine B detected in step 45, and the entity detected in step 47. A URI for feature data of entity B is generated from the combination of names of tracking methods. Subsequently, the AR processing unit 12 obtains feature data representing the characteristics of entity B in a format processable by the AR engine B using this URI.

Subsequently, the AR processing unit 12 executes the AR engine B based on the feature data obtained in this way to track the entity B in the augmented reality space. More specifically, the AR processing unit 12 calls AR engine B, inputs feature data of entity B into it, and executes AR engine B. Subsequently, the AR engine B returns to the AR processing unit 12 the result of tracking the entity by the AR engine B, for example, the location of the entity B to the AR processing unit 12 . Next, the AR processing unit 12 overlaps the virtual image of each entity B at the position of entity B with respect to each of a plurality of real space images input from the camera module 17 in real time, thereby generating an augmented reality image for each real space image. generate

When the augmented reality application N is executed and the detected entity is “entity N” and the properties of the AR engine N are set to be executed based on the original image of the entity N, steps 41 to 46 and 410 to 412 are performed. Entity N is tracked. That is, the URI processor 13 generates a URI for an original image of entity N from a combination of the host name of the resource server 2 and the name of entity N among at least one entity detected in step 43 . Subsequently, the AR processing unit 12 acquires the original image of entity N using this URI.

Next, the AR processing unit 12 executes the AR engine N based on the original image obtained in this way to track the entity N in the augmented reality space. More specifically, the AR processing unit 12 calls AR engine N, inputs the original image of entity N into it, and executes AR engine N. Subsequently, the AR engine N returns to the AR processing unit 12 the entity tracking result by the AR engine N, for example, the location of the entity N, to the AR processing unit 12. Then, the AR processing unit 12 overlaps the virtual image of each entity N at the position of the entity N with respect to each of a plurality of real space images input from the camera module 17 in real time, thereby generating an augmented reality image for each real space image. generate The execution process of other augmented reality applications is also performed according to any one of the above two examples.

FIG. 7 is a diagram showing an example of a URI table stored in the resource server 2 shown in FIG. 1 . The URI table shown in FIG. 7 is a database of feature data layered and stored according to a plurality of entities, a plurality of AR engines, and a plurality of entity tracking methods using a URI scheme, and is visualized to aid understanding. Since each URI of the URI table shown in FIG. 7 indicates where in the database of the resource server 2 the feature data or original image of an entity is stored, the AR client 1 quickly provides the feature data or original image of each entity. The original image can be obtained. According to the URI table shown in FIG. 7 , multiple types of AR engines are classified into "ARCore", "ARKit", "Vuforia", and "Wikitude".

As mentioned above, if an AR engine's properties are set to always perform the process of generating its feature data from an entity's original image before tracking it, then that AR engine will require the original image, not the feature data. . In this case, if the name of the entity detected in step 43 is "house", the URI for the original image of this entity may be "http://server.com/outdoor/house/OriginalImage". If the name of the entity detected in step 43 is "cat", the URI for the original image of this entity may be "http://server.com/indoor/cat/OriginalImage".

Since the original image of an entity is the same regardless of the type of each AR engine, it is preferable to unify the URI for the original image for each entity in terms of efficient use of the data storage space of the resource server 2. When the number of entities managed by the resource server 2 becomes very large, the URI for each entity can be hierarchically defined by grouping several entities for quick access to the storage location of each entity's feature data or original image. there is. That is, the URI of an entity may be generated by adding the entity name to the name of a group to which the entity belongs.

"ARCore" is a native AR engine that can only run on the Google Company's Android platform. It has a "Motion tracking" function that estimates the camera position in real space, an "Environment understanding" function that estimates the geometry of the real space, and a two-dimensional image tracking function. It provides "Augmented images" function, etc. "ARCore" does not provide the ability to track 3D objects.

If the 2D image shown in FIG. 2 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/outdoor/house". If the name of the AR engine detected in step 45 is "ARCore" and the name of the entity tracking method detected in step 47 is "AugmentedImages", the URI of the feature data generated in step 48 is "http://server.com/ outdoor/house/ARCore/AugmentedImages". This example applies when the property of "ARCore" is set to run based on the entity's feature data. If the "ARCore" property is set to be executed based on the original image of the entity, the URI of the original image as described above may be used.

"ARKit" is a native AR engine that can only run on Apple Company's iOS platform. "ARWorldTrackingConfiguration" function to estimate camera position in real space, "ARGeoTrackingConfiguration" function to estimate camera position using GPS, and 2D image tracking function It provides "ARImageTrackingConfiguration" function and "ARObjectScanningConfiguration" function that tracks 3D objects.

If the 2D image shown in FIG. 2 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/outdoor/house". If the name of the AR engine detected in step 45 is "ARKit" and the name of the entity tracking method detected in step 47 is "ARImageTracking", the URI of the feature data generated in step 48 is "http://server.com/ outdoor/house/ARKit/ARImageTracking".

If the 3D object shown in FIG. 3 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/indoor/cat". If the name of the AR engine detected in step 45 is "ARKit" and the name of the entity tracking method detected in step 47 is "ARObjectScanning", the URI of the feature data generated in step 48 is "http://server.com/ indoor/cat/ARKit/ARObjectScanning". These two examples correspond to cases in which the property of "ARKit" is set to run based on the entity's feature data. If the property of "ARKit" is set to be executed based on the original image of the entity, the URI of the original image as described above may be used.

"Vuforia" is PTC Company's AR engine running on both Android and iOS. "Area Targets" function to estimate camera position in real space, "Image Targets" function to track 2D images, 3D object tracking It provides "Model Targets" function, etc.

If the 2D image shown in FIG. 2 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/outdoor/house". If the name of the AR engine detected in step 45 is "Vuforia" and the name of the entity tracking method detected in step 47 is "ImageTargets", the URI of the feature data generated in step 48 is "http://server.com/ outdoor/house/Vuforia/Image Targets".

If the 3D object shown in FIG. 3 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/indoor/cat". If the name of the AR engine detected in step 45 is "Vuforia" and the name of the entity tracking method detected in step 47 is "Model Targets", the URI of the feature data generated in step 48 is "http://server.com". /indoor/cat/Vuforia/ModelTargets". These two examples correspond to the case where the property of "Vuforia" is set to run based on the entity's feature data. If the property of "Vuforia" is set to be executed based on the original image of the entity, the URI of the original image as described above may be used.

"Wikitude" is the Wikitude company's AR engine that runs on both Android and iOS. It has a "Scene Tracking" function that estimates the camera position in real space, and "Instant" that can overlap virtual images by recognizing planes. Tracking" function, "Image Tracking" function for tracking 2D images, and "Object Tracking" function for tracking 3D objects are provided.

If the 2D image shown in FIG. 2 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/outdoor/house". If the name of the AR engine detected in step 45 is "Wikitude" and the name of the entity tracking method detected in step 47 is "ImageTracking", the URI of the feature data generated in step 48 is "http://server.com/ outdoor/house/Wikitude/ImageTracking".

If the 3D object shown in FIG. 3 is the entity detected in step 43, the URI of the entity created in step 44 becomes "http://server.com/indoor/cat". If the name of the AR engine detected in step 45 is "Wikitude" and the name of the entity tracking method detected in step 47 is "ObjectTracking", the URI of the feature data generated in step 48 is "http://server.com/ indoor/cat/Wikitude/ObjectTracking". These two examples are when the property of "Wikitude" is set to run based on the entity's feature data. If the property of "Wikitude" is set to be executed based on the original image of the entity, the URI of the original image as described above may be used.

As described above, not only does the function provided differ according to the type of each AR engine, but also the entity tracking performance of each AR engine differs depending on whether the entity corresponds to a 2D image or a 3D object, and depending on the shape of the entity. Also, the entity tracking performance of each AR engine is different. Therefore, it is necessary to design an augmented reality application so that an AR engine suitable for augmented reality content is provided to the user according to what kind of augmented reality service is provided. Augmented reality applications may use a plurality of different AR engines to provide high-quality augmented reality services. In particular, since “ARCore” and “ARKit” can be executed only on their dedicated platforms, separate augmented reality applications need to be designed for the same augmented reality service.

According to the present embodiment, the feature data of the entity is obtained from the resource server 2 using the URI for the feature data of the entity independently of the AR engine and the entity tracking method, and AR is performed based on the feature data obtained in this way. Since the engine runs, the augmented reality application does not need to include various algorithms that generate feature data from the original image of the entity for each AR engine and entity tracking method, enabling the development of an efficient augmented reality application.

Entity tracking in the augmented reality space according to the present embodiment can be achieved only by obtaining the entity's feature data from the resource server 2 using the URI for the entity's feature data, and calling and executing the AR engine. Therefore, it is possible to provide a general-purpose augmented reality application that does not include an AR engine and a feature data generating algorithm dependent on an entity tracking method.

As the augmented reality application does not need to include feature data generating algorithms for each AR engine and entity tracking method, the size of the augmented reality application can be reduced, and thus the amount of work required to develop the augmented reality application can be reduced. When developing any augmented reality application, most of the algorithms, such as the algorithm for obtaining feature data of the entity from the resource server (2) using the URI of the entity's feature data in the existing augmented reality application, can be reused as they are. Development of real-world applications becomes very easy.

Meanwhile, the entity tracking method according to an embodiment of the present invention as described above can be written as a program executable on a computer processor, and can be implemented on a computer that records and executes the program on a computer-readable recording medium. there is. The computer includes all types of computers capable of executing programs, such as desktop computers, notebook computers, smart phones, and embedded-type computers. In addition, the structure of data used in one embodiment of the present invention described above can be recorded on a computer-readable recording medium through various means. Computer-readable recording media include storage such as RAM, ROM, magnetic storage media (eg, floppy disk, hard disk, etc.), optical reading media (eg, CD-ROM, DVD, etc.) includes media

So far, the present invention has been looked at with respect to its preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

1 ... AR client
11 ... main processor
12 ... AR processing unit
13 ... URI processing unit
14 ... communication module
15 ... input module)
16 ... output module
17 ... camera module
18 ... sensor module
19 ... Storage
20 ... bus
2 ... resource server

Claims (11)

In the feature data-based entity tracking method independent of the augmented reality engine,
generating a Uniform Resource Identifier (URI) for feature data of the first entity from a combination of a name of the first entity and a name of the first AR engine;
Displaying features of the first entity in a format processable by the first AR engine to a resource server corresponding to a host of the URI for feature data of the first entity using the URI for feature data of the first entity. obtaining feature data of the first entity from the resource server by requesting feature data of the first entity; and
and performing tracking of the first entity in an augmented reality space by executing the first AR engine based on the acquired feature data of the first entity. According to claim 1,
The generating of the URI for feature data of the first entity may include a URI for feature data of the first entity from a combination of the host name of the resource server, the name of the first entity, and the name of the first AR engine. Entity tracking method characterized in that for generating. According to claim 2,
generating a URI of the first entity from a combination of the host name of the resource server and the name of the first entity; and
Further comprising detecting a first AR engine to be used for tracking the first entity from among multiple types of AR engines;
The generating of the URI for the feature data of the first entity includes generating the URI for the feature data of the first entity by adding the name of the detected first AR engine to the generated URI of the first entity. Characterized entity tracking method. According to claim 3,
Further comprising detecting an entity tracking method to be used by the first AR engine for the first entity;
The generating of the URI for the feature data of the first entity may include adding the name of the detected first AR engine and the name of the detected entity tracking method to the URI of the first entity to obtain information about the first entity. An entity tracking method characterized by generating a URI for feature data. According to claim 1,
The detecting of the entity tracking method includes detecting an entity tracking method to be used by the first AR engine for the first entity according to whether the first entity corresponds to a 2D image or a 3D object. Characterized entity tracking method. According to claim 1,
Further comprising the step of detecting at least one entity in the augmented reality space represented by the augmented reality content from the augmented reality content,
In the generating of the URI for the feature data of the first entity, the URI for the feature data of the first entity is obtained from a combination of a name of the first entity among the detected at least one entity and a name of the first AR engine. An entity tracking method characterized by generating a URI. According to claim 1,
generating a URI for feature data of a second entity from a combination of a name of a second entity and a name of a second AR engine;
acquiring feature data of a second entity representing features of the second entity in a format processable by the second AR engine by using a URI of the feature data of the second entity; and
and performing tracking of the second entity in an augmented reality space by executing the second AR engine based on the acquired feature data of the second entity. According to claim 2,
The first AR engine may further include determining which of feature data or an original image of the first entity is required for tracking the first entity,
The generating of the URI for the feature data of the first entity includes generating a URI for the feature data of the first entity if it is determined that the first AR engine requires the feature data of the first entity. Characterized entity tracking method. According to claim 8,
If it is determined that the first AR engine requires the original image of the first entity, generating a URI for the original image of the first entity from a combination of a host name of a resource server and the name of the first entity. ;
acquiring an original image of the first entity by using a URI of the original image of the first entity; and
and performing tracking of the first entity in the augmented reality space by executing the first AR engine based on the acquired original image of the first entity. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 9 in a computer. In the feature data-based entity tracking device independent of the augmented reality engine,
a URI processor generating a URI for feature data of the first entity from a combination of the name of the first entity and the name of the first AR engine; and
Displaying features of the first entity in a format processable by the first AR engine to a resource server corresponding to a host of the URI for feature data of the first entity using the URI for feature data of the first entity. The feature data of the first entity is acquired from the resource server by requesting feature data of the first entity, and the first AR engine is executed based on the obtained feature data of the first entity to obtain the first entity in the augmented reality space. An entity tracking device comprising an AR processor that tracks an entity.

Images