KR102232408B1 - Augmented Reality system for advertising platform based image analysis - Google Patents

Abstract

An embodiment of the present invention is a terminal used by an advertiser, comprising: an advertiser terminal in which an augmented reality advertising platform app is installed for registering advertisement content to be displayed based on a marker and an augmented reality target; A terminal used by a user, comprising: a user terminal installed with an augmented reality advertising platform app that scans a marker in an area where the user is located and receives and executes an augmented reality targeting the scanned marker from an augmented reality server; An augmented reality server that generates and stores an augmented reality in which advertisement content is displayed based on the marker registered by the advertiser, and extracts an augmented reality based on the marker requested by the scan of the user terminal and provides it to the user terminal; And a terminal used by an administrator managing the augmented reality server, wherein a management UI including user authentication and personal information management UI of the augmented reality server, marker information management UI, content model information management UI, and item purchase information management UI is provided. It may include; provided manager terminal.

Description

Augmented Reality system for advertising platform based image analysis through image analysis-based marker production

The present invention relates to an advertisement platform system based on an augmented reality through image analysis-based marker production as an advertisement platform system.

In recent years, along with the 4th industrial revolution, the availability of technologies such as augmented reality/virtual reality is increasing, and the size of the related market is increasing.

In the case of augmented reality, it refers to a technology that creates a virtual object on the original input image and outputs a new image synthesized with the original image. As a representative example of such a technology, advertisements using augmented reality technology in sports commentary broadcasts, information related to players or games are provided, and in recent years, it is also used in various ways in mobile applications for educational purposes.

In the field of augmented reality, according to a research study by Digi-Capital, foreign companies are showing various devices and contents centering on virtual reality (VR), but in the industrial field, augmented reality (AR) is used in various fields. The observation that it will be used and will become a key axis prevail, and the augmented reality market is expected to increase from $1 billion in 2016 to $1,200 in 2020, becoming a relatively larger market than virtual reality.

The biggest problem in commercializing augmented reality technology is that there is a problem that an environment in which augmented reality technology can be used easily and quickly is lacking, and content accordingly is not sufficient.

In addition, there are not many platforms in which a user who registers such augmented reality content can directly manage or manipulate, and accordingly, augmented reality technology is rarely commercialized in the advertising market, and practicality is not secured.

On the other hand, in the case of small-scale self-employed, it is very difficult to compete with medium-sized and large corporations in terms of various advertisements, publicity costs, and originality. There are parts that are not practically suitable for small or individual to use the general business models frequently used by large companies for advertising or publicity purposes, which is an important cause of lowering the competition rate and closing.

Korean Patent Publication 10-2009-0110357

The technical problem of the present invention is to maintain a stable platform by solving the problem of registering markers and managing and maintaining markers on the platform itself in constructing an advertising platform system based on augmented reality. have.

In addition, when an advertiser registers a marker, it gives feedback to the registrant and improves the stability of the platform by assigning an evaluation score for the marker to be registered in order to determine whether the image to be registered can function normally as a marker. There is a purpose in trying to make it happen.

In addition, when a random user scans a marker within a specific radius of the location where the marker is registered, the augmented reality content related to the marker is provided by providing correct information about the marker to the user by searching for the marker most similar to the image and filtering. The purpose is to provide and make it available for advertising.

An embodiment of the present invention is a terminal used by an advertiser, comprising: an advertiser terminal in which an augmented reality advertising platform app is installed for registering advertisement content to be displayed based on a marker and an augmented reality target; A terminal used by a user, comprising: a user terminal installed with an augmented reality advertising platform app that scans a marker in an area where the user is located and receives and executes an augmented reality targeting the scanned marker from an augmented reality server; An augmented reality server that generates and stores an augmented reality in which advertisement content is displayed based on the marker registered by the advertiser, and extracts an augmented reality based on the marker requested by the scan of the user terminal and provides it to the user terminal; And a terminal used by an administrator managing the augmented reality server, wherein a management UI including user authentication and personal information management UI of the augmented reality server, marker information management UI, content model information management UI, and item purchase information management UI is provided. It may include; provided manager terminal.

The augmented reality advertising platform app includes a user authentication menu for authenticating a user or advertiser, a marker and business card registration menu for generating augmented reality advertising content, a marker scan menu for using augmented reality advertising content, and augmented reality advertising content It may include a purchase menu of advertisement items that can be used for production.

When user authentication is successful through the user authentication menu, user location information at the time of authentication may be transmitted to and stored in an augmented reality server.

The purchase menu of advertisement items that can be used to produce augmented reality advertisement contents includes a user interface (UI) that displays an advertisement item list, which is a list of pre-registered advertisement items, and then makes a purchase of an advertisement item selected by an advertiser. It can be characterized by providing.

The manager terminal may be characterized in that the advertisement item list can be updated and managed through the content model information management UI.

The marker and business card registration menu for generating the augmented reality advertisement content is a user interface (UI) that allows an advertiser to register a marker including a specific photo and menu board in his or her store through an album or a camera of the advertiser terminal. Wow, it may be characterized by providing a user interface (UI) in which the advertiser selects location information of a marker to be registered by displaying a map.

The marker and business card registration menu for generating the augmented reality advertisement content, if the marker to be registered is an advertiser business card, a business card phone instead of the location information of the marker so that the marker can be identified using an identification number capable of recognizing the business card. It may be characterized by receiving a number.

The marker and business card registration menu for generating the augmented reality advertisement content may be characterized by providing a user interface (UI) for making a purchase of advertisement content selected by an advertiser after displaying a list of advertisement contents. .

The augmented reality server calculates and provides the evaluation value of the marker registered by the advertiser, and the marker and business card registration menu for generating the augmented reality advertisement content receive and display the evaluation value of the marker registered by the advertiser from the augmented reality server. It may be characterized in that it provides a user interface (UI).

The evaluation value of the marker is determined by extracting the feature point of the registered marker and grasping the position coordinate, size, and image change direction value at which the feature point is located, and determines whether the position coordinate, size, and image change direction value are within a set reference position range. It can be characterized as an evaluated value.

The augmented reality server generates and stores an augmented reality in which advertisement content is displayed based on the registered marker. When the selection of the marker and advertisement content is completed by the advertiser, the marker recognition and tracking, visualization test and scale adjustment, and user UI are performed. It may be characterized by generating and storing an augmented reality that displays advertisement content based on a marker after designating a rotation value for it.

The scale adjustment and designation of the rotation value for the user UI are set by the advertiser using the navigation bar and arrow buttons displayed on the advertiser terminal, and in the case of setting the scale, drag the navigation bar to increase the size in steps of 0 to 100. It is set to any one value within the range, and in the case of setting the rotation value, it is divided into X-axis, Y-axis, and Z-axis. It is characterized in that the direction of advertisement content is set by rotating by 45 degrees by clicking the arrow button. I can.

According to an embodiment of the present invention, it is possible to provide efficient content information by providing dynamic content information on a place with regional settings through the production of augmented reality content, and to attract continuous tourists by inducing interest in users. Can be prepared.

In addition, according to an embodiment of the present invention, the effect of reducing marketing costs of small businesses is expected due to the recent improvement in demand through easy augmented reality content production, and a low-cost, high-efficiency content service can serve as a medium for revitalization of local commercial areas.

1 is a diagram showing the functional structure of an augmented reality-based advertising platform through image analysis-based marker production according to an embodiment of the present invention.
2 is a diagram showing each functional module of an augmented reality-based advertising platform system through image analysis-based marker production according to an embodiment of the present invention.
3 is a block diagram of an augmented reality-based advertisement platform system through image analysis-based marker production according to an embodiment of the present invention.
4 is a use case diagram of a management service according to an embodiment of the present invention.
5 is an exemplary screen of a management UI according to an embodiment of the present invention.
6 is an exemplary screen of a content management service database UI according to an embodiment of the present invention.
7 is an exemplary screen of a management service Storage UI according to an embodiment of the present invention.
8 is an exemplary screen of an in-app product management UI according to an embodiment of the present invention.
9 is an exemplary screen of an in-app product order management UI according to an embodiment of the present invention.
10 is a use case diagram of an augmented reality advertising platform app according to an embodiment of the present invention.
11 is a use case diagram of a location information service of an augmented reality advertising platform app according to an embodiment of the present invention.
12 is an advertisement item store purchase service use case diagram of an augmented reality advertisement platform app according to an embodiment of the present invention.
13 is a diagram showing a state in which advertisement item content list inquiry, purchase, and payment are made according to an embodiment of the present invention.
14 is an exemplary diagram of a marker registration use case diagram according to an embodiment of the present invention.
15 is an exemplary screen illustration of a marker registration UI according to an embodiment of the present invention.
16 is a flowchart showing a marker registration algorithm according to an embodiment of the present invention.
17 is a use case diagram for marker recognition, tracking, and visualization required to experience augmented reality content according to an embodiment of the present invention.
18 is an exemplary screen of a marker and business card search UI according to an embodiment of the present invention.
19 is an exemplary diagram of an augmented reality GUI screen according to an embodiment of the present invention.
20 is a diagram showing a marker test algorithm according to an embodiment of the present invention.
21 is a diagram showing a marker evaluation algorithm according to an embodiment of the present invention.

Hereinafter, advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and is provided to completely inform the scope of the invention to those of ordinary skill in the art to which the present invention pertains. As such, the invention is only defined by the scope of the claims. In addition, in describing the present invention, when it is determined that related known technologies may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is a diagram showing the functional structure of an augmented reality-based advertising platform through image analysis-based marker production according to an embodiment of the present invention, and FIG. 2 is augmented reality through image analysis-based marker production according to an embodiment of the present invention. Figure 3 is a diagram showing each functional module of the based advertising platform system, Figure 3 is a configuration diagram of an augmented reality-based advertising platform system through image analysis-based marker production according to an embodiment of the present invention, Figure 4 is an embodiment of the present invention 5 is an exemplary screen of a management UI according to an embodiment of the present invention, and FIG. 6 is an exemplary screen of a content management service database UI according to an embodiment of the present invention, and FIG. 7 Is an example screen of a management service Storage UI according to an embodiment of the present invention, FIG. 8 is an example screen of an in-app product management UI according to an embodiment of the present invention, and FIG. 9 is an in-app product order management according to an embodiment of the present invention. An example screen of a UI, FIG. 10 is a use case diagram of an augmented reality advertisement platform app according to an embodiment of the present invention, and FIG. 11 is a use case diagram of a location information service of an augmented reality advertisement platform app according to an embodiment of the present invention. 12 is an advertisement item store purchase service use case diagram of an augmented reality advertisement platform app according to an embodiment of the present invention, and FIG. 13 is an advertisement item content list inquiry, purchase, and payment according to an embodiment of the present invention. Fig. 14 is an exemplary picture of a marker registration use case diagram according to an embodiment of the present invention, Fig. 15 is an exemplary screen picture of a marker registration UI according to an embodiment of the present invention, and Fig. 16 is It is a flowchart showing a marker registration algorithm according to an embodiment of the present invention, and FIG. 17 is a use case diagram for marker recognition, tracking, and visualization required to experience augmented reality content according to an embodiment of the present invention. An exemplary screen of a marker and business card search UI according to an embodiment of the present invention, and FIG. 19 is an augmented reality GUI according to an embodiment of the present invention. Fig. 20 is a diagram showing a marker test algorithm according to an embodiment of the present invention, and Fig. 21 is a diagram showing a marker evaluation algorithm according to an embodiment of the present invention.

The advertisement platform system of the present invention provides a user with an environment in which image recognition-based augmented reality content can be directly produced, as shown in FIG. 1, thereby creating augmented reality markers, tracking algorithms, and recognition image processing. It provides a smartphone app UI that can develop algorithms and produce augmented reality contents.

Accordingly, the augmented reality-based advertisement platform system of the present invention can serve as a platform through which advertisers and general users can connect with each other through augmented reality content. Management system module that can manage and control the augmented reality platform as a whole, mobile application module that users can access to experience augmented reality, augmented reality module that can operate in a mobile environment, augmented reality module implementation and marker management An image processing module, an item store for linking with an advertiser, and a demonstration content production module may be provided.

To this end, the advertisement platform system of the present invention includes a wired/wireless communication network (not shown), a user terminal 100, an administrator terminal 300, an advertiser terminal 200, and an augmented reality server 400 as shown in FIG. 3. Includes.

A wired/wireless communication network (not shown) provides wired communication or wireless communication between the user terminal 100, the manager terminal 300, the advertiser terminal 200, and the augmented reality server 400.

The user terminal 100 is a terminal used by a user, and may correspond to a communication terminal such as a smartphone. The user terminal 100 is installed with an augmented reality advertisement platform app that scans a marker in an area where the user is located and receives and executes an augmented reality targeting the scanned marker from the augmented reality server 400.

For reference, the user terminal 100 and the advertiser terminal 200 implemented as a smart phone, etc., can install and add or delete hundreds of various applications (application programs) as desired by the user. You can also create the application you want, and you can implement an interface suitable for you through various applications. Therefore, you can download an augmented reality advertising platform app from Google Market and Apple Store and install it on your smartphone.

The manager terminal 300 is a terminal used by an administrator who manages the augmented reality server 400, and may correspond to a communication terminal such as a desktop PC or a smartphone.

The manager terminal 300 is a management UI including user authentication and personal information management UI of the augmented reality server 400, marker information management UI, content model information management UI, and item purchase information management UI, as shown in FIG. 4. Is provided.

To this end, the management UI (User Interface) is a cloud-based server system UI for an augmented reality-based advertisement platform service. The cloud service uses Google's cloud platform. It creates a Compute Engine instance to configure cloud computing, and connects the cloud database and cloud storage to store data. In order to link with the mobile service of an augmented reality-based advertising platform, Firebase, an open platform, is linked. The Firebase module makes it easy to integrate various services from Google Cloud with mobile platforms. For payment services for augmented reality-based advertising platform services, the in-app payment function of the Google Play app can be added. Through the in-app payment service, users can easily purchase in-app products without the need to additionally execute a separate payment module.

Therefore, the management service through the management UI can add a user, check user information, reset a password, stop using an account, and delete an account. The content model information management service may register information of an augmented reality content model, modify content model information, and delete content model information. Item purchase information management can register information on items sold through the app, modify item information, delete item information, and check purchase history information purchased by users through the app.

For example, the management UI can check the user's subscription type, subscription date, last login date, user e-mail, user-specific identification UID value through a list of users who have subscribed to the augmented reality advertisement platform system as shown in FIG. 5. And, it can handle the functions of resetting the password of the user account, changing the account activation status, and deleting the account. User registration can be made through membership registration of the augmented reality advertising platform mobile application, or users can be added or deleted through the user management service UI of the management system.

In addition, the advertisement item list can be updated and managed through the content model information management UI. That is, as shown in FIGS. 6 and 7, content information is managed in units of a document of a content model, and is composed of a content ID, a description, a file name of the model, a name, a product ID, a texture file name, and a thumbnail file name. When registering content information, after uploading the content configuration file to the storage service, you can add the document by entering the model information.

In addition, as shown in FIGS. 8 and 9, the management UI registers a sale product by inputting a product ID, a product name, a product description, an activation state, and price information to register an in-app product. You can use the translation add function to provide automatic translation information according to the user's locale. When you register a product, you can check the list of registered products. You can check the sales history of the item through the order management menu.

The advertiser terminal 200 is a terminal used by an advertiser and may correspond to a communication terminal such as a smartphone or a desktop PC.

The advertiser terminal 200 has an augmented reality advertisement platform app installed, and the augmented reality advertisement platform app installed on the advertiser terminal 200 receives a marker that is an augmented reality target and an advertisement content to be displayed based on the marker.

When the terminal logs in to the advertiser through the augmented reality advertising platform app, it plays the role of the advertiser terminal 200, and when the user logs in through the augmented reality advertising platform app, it plays the role of the user terminal 100.

As shown in FIG. 10, this augmented reality advertisement platform app includes i) a user authentication menu that performs user or advertiser authentication, ii) a purchase menu of advertisement items that can be used to produce augmented reality advertisement content, and iii) augmented reality. A marker and business card registration menu for generating advertisement content, and iv) a marker scan menu for using augmented reality advertisement content. It will be described in detail below.

(1) User authentication menu UI design

The user or advertiser must first be authenticated as a user or advertiser through the user management service.

For example, if the user is not registered on the augmented reality advertising platform, validateForm(), a validity function for entering personal information such as email, password, and name through the membership registration activity UI, and checking the validity of the entered values. Call the function. If the input value is invalid, an error is displayed on the form. If the value is valid, the progress screen is displayed through the showProgress function, and then the createUserEmailAndPassword() function is called, and the user information value entered by the user is delivered. If the member registration request is successful through the user management service, the AuthResult value is received, and if it fails, the error message is checked through the getException() function. If the request to create a user is successful, an e-mail confirmation procedure is performed to activate the rights of the user who has signed up for membership. The sendEmailVerification() function is called to retrieve the unique information of the subscribed user through the getCurrentUser() function and to request an email confirmation for the user. When the confirmation e-mail is successfully sent to the user, information on the user who has signed up is stored in the database. Call the getUid() function to receive the user's unique userid value through the user management service of the management system, and check the e-mail and name information entered by the user, Location information indicating the user's current location, and whether to use the user's notification message. Configures the indicated notification information as a UserModel class instance and transfers it with the requestSetUserInfo() function to request database storage. After receiving the response to the request to save the database, the membership registration activity is terminated and the login activity is moved.

On the other hand, when user authentication is successful through the user authentication menu, the terminal is implemented as the user terminal 100, and as shown in FIG. 11, the user location information at the time of authentication is transmitted to the augmented reality server 400 and stored. . The stored user location information is used to generate augmented reality.

For example, the augmented reality advertising platform app calls the getLastLocation() function to get the latest location information of the mobile device when the updateUI() function of the main activity is called. If the request for the getLastLocation() function of the FusedLocationProviderClient class is successful, a Location class instance containing geographic location information is returned, and the location information value of the current device can be checked through values such as latitude, longitude, altitude, and speed contained in the Location class. have. In augmented reality advertising platform mobile application, the FetchAddressIntentService service class is used to convert this geographic-based location information into address information to make it easier to identify. Address information Request address information by passing geographic location information Location class instance to address information service class, calling getFromLocation() function of Geocoder class and passing latitude and longitude values. If the request for address information is successful, the Address class instance containing the address value is returned in the language according to the locale setting, and the address value is responded to the main activity that called the service. When clicking the real-time location button in the main activity, the requestLocationUpdates() function of the LocationManager class is called, and LocationListener, GPS_PROVIDER, NETWORK_PROVIDER, minTime (in milliseconds), and minDistance (in meters) values are passed. Receives a Location class instance containing real-time location information as a response through LocationListener according to the specified time and interval. In the augmented reality advertising platform mobile application, a certain range of areas can be designated based on geographic location information, and status information about when entering/exiting the area and whether it is within the range can be provided through a notification message. First, prepare a GeofencingClient client instance by calling the getGeofencingClient() function of the LocationServices class to set the area. Create a Geofence instance by designating the location information to be the reference point as latitude, longitude, and radius values, call the addGeofences() function of the GeofencingClient class, and designate a listener to receive a response. If the area designation request is successful, the response value is handled through the previously registered GeofenceTransitionsIntentService service class. The event for the region is determined through the state values of GEOFENCE_TRANSITION_ENTER, GEOFENCE_TRANSITION_EXIT, and GEOFENCE_TRANSITION_DWELL. In the augmented reality advertising platform mobile application, when an area event occurs, a NotificationChannel is created by calling the sendNotification() function, and when a user selects a notification message, the app is executed and a specific activity is executed by designating an activity class to create a NotificationCompat. In the case of real-time location information, a button is created and provided so that the user can easily select the usage because the battery usage increases.

(2) UI design of the purchase menu of advertisement items that can be used to produce augmented reality advertisement contents

As shown in Figs. 12 and 13, the purchase menu of advertisement items that can be used to produce augmented reality advertisement contents displays an advertisement item list, which is a list of pre-registered advertisement items, and then purchases advertisement items selected from advertisers. It is designed to provide a user interface (UI) that allows the operation to be performed. Each advertisement content list shows the contents added to the in-app purchase list of the manager server and the advertisement contents registered in the database, and the user purchases the desired advertisement contents through the purchase of advertisement contents, and is then augmented when a marker is registered. Real content can be used.

For example, advertisement items registered through the item store management service may be moved from the side menu of the main activity to the item list activity through the AR Contents menu. The advertisement item list consists of each item item containing sku_id, title, price, description, currency_code, and billing_type information through the SkuDetailModel class. When the user clicks the purchase button of the desired line item item, the purchaseItem() function of the PurchaseManager class is called, and the skuid value and skutype value of the item are delivered. When the purchase request is completed, a response is received with a list of Purchase class instances whose purchase information has been updated. When the purchase is successful, the requestSetContentInfo() function is called to update the purchase information of the advertisement content through the database management service of the management system. When the update request is completed, the newly obtained item information is updated in the advertisement item item held by the user, and the list is updated. The advertisement item items held by the user are used as augmented reality content and are selected and used when creating an augmented reality advertisement service.

(3) Marker and business card registration menu UI design for creating augmented reality advertisement content

Referring to the use case diagram of marker registration shown in Fig. 14, the advertiser must register a marker he intends to use, such as a specific photo in his store or a menu board. If the marker to be registered is an advertiser's business card, a business card phone number is input instead of the location information of the marker so that the marker can be identified using an identification number capable of recognizing the business card.

For example, the image you want to use as a marker must be able to import a photo from an album on your mobile or take it on the spot and import it in your local environment. The marker registrant is the first step, taking an image from a capture from an album or camera to get an image to use as a marker. Afterwards, the area to be used as a marker must be cropped and given as input. In addition, by giving an image as an input to the marker evaluation module, the registrant should be able to receive the predicted value from the evaluation module on how much performance the corresponding marker has. In the case of the marker evaluation module, the image and userID must be registered while sending a request to the image processing module. In the next step, the registrant needs to know the location information (longitude, latitude) value of the marker to be registered, so to easily get the location where the marker to be registered exists, display the map using the google map api and mark it. You must be provided with a UI for entering the location. When the marker to be registered is a business card, analysis by extracting feature points from the image is meaningless, and thus the marker is identified using an identification number capable of recognizing the business card instead of location information. Therefore, registration proceeds through the process of entering the phone number of the business card to be registered.

Furthermore, in the step of selecting a content, since the marker registrant must select the content to be displayed on the actual marker, it must be provided which content is owned. Augmented reality content included in the selection list is largely classified into image, video, and 3D content. In order to select a content, the content can be purchased through the item store, and the selectable content is displayed in a list according to the purchase history. The marker registrant selects the content to be actually applied from the provided possession content, and finally must be able to confirm the marker to be registered and the part of the content to be displayed. Since the marker registrant may want the image size of the registered marker or the content to be displayed to be expressed differently, UI is provided to set whether or not the content actually works well on the marker, and the size and direction of the displayed content. Should be. For detailed content setting, the corresponding values can be adjusted in the stage of testing the marker, and in the case of scale, it can be determined by adjusting the scale value between 0 and 100. In the case of rotate, the x, y, z axes can be adjusted. Each direction is adjusted and set. Finally, when the marker test is finished, it is finally uploaded through the platform's management system server, and the marker image and information on detailed model settings are uploaded together and finally stored in the database and storage linked with the platform's management system. The marker is finally registered.

Therefore, as shown in FIG. 15, a user interface (UI) and a map through which an advertiser can register a marker including a specific photo and menu board in his or her store through an album or a camera of the advertiser terminal 200. A user interface (UI) in which the advertiser selects location information of a marker to be registered by the advertiser is provided. The mobile marker registration GUI is largely classified into areas for selecting a marker image, selecting an augmented reality content, and checking marker registration information. In the portion of selecting an image to be used as a marker, selection for registration of a general marker and a business card marker, selection of an image to be used as a marker by clicking the ImageViw portion, and a marker evaluation result through the augmented reality server 400 can be known. Second, in the location information selection part, it is possible for the user to select an exact location through drag or search for a desired location by address and adjust the location information value. Third, it is possible to receive and select information on the augmented reality content held by the user. In addition, in the step of finally checking the marker registration information entered in the previous step in relation to the marker registration, preview the image of the marker to be registered, check the evaluation score, check the marker registration location information, and the augmented reality content obtained as a result of the marker test. You can check the size and rotation values again.

Further, the marker and business card registration menu for generating augmented reality advertisement content provides a user interface (UI) for receiving and displaying the evaluation value of the marker registered by the advertiser from the augmented reality server 400. The evaluation value of such a marker is determined by extracting the feature point of the registered marker and grasping the position coordinate, size, and image change direction value at which the feature point is located, and determines whether the position coordinate, size, and image change direction value are within a set reference position range. It is an evaluated value.

As a result, the marker registration algorithm, as shown in Fig. 16, undergoes a process of selecting an image of a marker to be registered, evaluating a marker, selecting a marker, storing identification information, selecting an advertisement content, checking basic marker registration information, and estimating the scale and rotation values. Markers can be registered.

For example, as shown in FIG. 17, in order to select an image to be registered as a marker, a gallery app or a camera app installed on a smartphone is accessed. In order to access the function, the application needs to be authorized through the manager class, and the flow chart assumes that the authorization processing part is approved. When an image is selected, an image for the selected area is regenerated through the manager class, and finally, an image to be used as a marker stored in the gallery is transmitted to the augmented reality server 400 to evaluate the marker. After the marker evaluation is completed and the result is reached, select the toggle button above to select whether to register as a general marker or an image to be used as a business card. After that, when registering as a general marker, it moves to the step of selecting the location information of the marker to be registered, and when registering a business card, the screen is switched to a screen for entering a mobile phone number or a unique identification number of the business card. . After that, after completing the input on each screen, it moves on to the next step. The next step is to select augmented reality content to be actually visualized on the marker, and if there is a history of purchasing or purchasing the content through the item store of the mobile application, it is displayed as a list so that the content can be selected. After that, by clicking the Next button, the marker registration information input in the previous step is finally output. After checking, click the Test Marker button to test the marker. In the testing phase, check whether the content is augmented properly, and specify values for the size and rotation of the content according to the image size of the marker to be registered. After that, click the Done button to return to the previous step, and finally click the Next Step button to register the marker according to the information to be registered. Each of the registered markers converts a marker image and information about the marker into a file format and stores them.

(4) Marker scan menu UI design for augmented reality advertisement content use

The marker scan menu UI design, as shown in FIG. 17, is a menu for marker recognition, tracking, and visualization required to experience augmented reality content. First, the user needs to approve the authority to use the camera in order to experience the augmented reality. In order to track a marker, a marker to be tracked and content information about it are required.

In addition, there is a need for a marker scan and search function to check whether an augmented reality marker that the user intends to experience is located near the user. For this, a marker and a business card search GUI as shown in FIG. 18 are provided. Augmented reality marker search GUI is largely divided into general image markers and business card markers. When the user touches the scan button while the marker to be searched is illuminated using the camera preview on the smartphone screen, the captured image is uploaded from the smartphone to the image processing server. The image processing server searches for a marker using the uploaded image and the uploaded location information value, and retrieves the marker database stored in the management server and searches whether there is an image similar to the uploaded image. If the matching is successful, the ID of the marker that successfully matched is returned, and the task of downloading the augmented reality content from the smartphone is proceeded. In the case of a business card, a request is made to the image processing server and the ID of the business card matching the phone number of the business card is returned, and the same operation as above is performed.

When the user touches the scan button through the application, it is done by scanning the marker. When the marker is scanned, the captured image is uploaded to the augmented reality server 400, and when uploading the image, the location information where the user is located is also uploaded. Based on the image and location information uploaded by the user, the augmented reality server 400 fetches data on candidate markers that can be experienced in the database based on the location information. It compares and finally returns information of the searched and matched markers to the user. According to the matched result, if the match is successful, the user brings a marker image for an augmented reality experience, a video or image of content linked with the registered marker, or a 3D model file. In addition, in the case of a 3D model, textures for the model can also be downloaded through the management server. When all of the tasks have been completed normally, after creating a new screen to experience the augmented reality, the augmented reality module is set. Depending on whether the augmented reality content is classified as a video, image, or 3D model, the module identifies and sets the module accordingly, and the user can experience the augmented reality content through marker recognition. When the marker is tracked normally, the vibration sensor of the smartphone is operated to transmit a signal indicating that the marker tracking was successful so that it can be recognized. In addition, the content size is adjusted so that the user can view the same screen as the value set by the registrant by reflecting the size, rotation, and animation values of the content set by the registrant who registered the marker.

If it fails to search for a marker, it is impossible to experience augmented reality because the content related to the marker cannot be imported.In this case, the user cannot obtain location information, or because the user does not approve the permission to access the location information sensor. This may correspond to a case in which the function does not work, and a case in which the scan item does not work because the camera authority is not approved. Even when the augmented reality server 400 and the management server are closed, the operation may not be performed properly. If you experience augmented reality for a registered business card, you can experience augmented reality likewise through the input of the identification number stored when the business card marker is registered by clicking the business card button next to the scan button.

Meanwhile, the augmented reality server 400 generates and stores an augmented reality in which advertisement content is displayed based on the marker registered by the advertiser, and extracts the augmented reality based on the marker requested by the scan of the user terminal 100 It is a server that performs a function provided to the user terminal 100. Accordingly, as illustrated in FIG. 19, augmented reality such as a video, image, and 3D model can be generated and displayed through the user terminal 100.

The augmented reality server 400 generates and stores an augmented reality in which advertisement content is displayed based on the registered marker, when marker and advertisement content selection is completed by the advertiser, marker recognition and tracking, visualization test and scale adjustment, and user After designating the rotation value for the UI, an augmented reality that displays advertisement content based on the marker is created and stored.

In the above, tracking under the marker recognition is possible by the augmented reality marker tracking by the module in which the setting is completed. When the user shines a marker on the Camera Preview screen, the user can check whether the marker tracking is performed normally with the vibration of the smartphone. Depending on the state of the marker, the tracking performance may be significantly deteriorated, which may vary depending on the marker evaluation score performed in the previous step. When the marker image to be registered has a repetitive pattern or feature points present in the marker are concentrated in one area by variance calculation, there is a possibility that the tracking performance may not operate properly due to slight obscuration or changes in lighting.

In addition, scale adjustment and designation of a rotation value for the user UI are set by the advertiser using the navigation bar and arrow buttons displayed on the advertiser terminal 200 to set the scale and rotation value. In the case of setting the scale, drag the navigation bar to set the size to 0. It is set to any one value within ~100 steps. In the case of setting the rotation value, it is divided into the X-axis, Y-axis, and Z-axis. Click the arrow button to rotate by 45 degrees to set the direction of the advertisement content. In detail, after the vibration, when the marker starts to be tracked normally, the augmented reality content is visualized on the marker. The visualized content has not yet been optimized for size and rotation values, but users can adjust the size and rotation values using the seekBar and arrow buttons on the screen. In the case of the same rotation value, it is divided into the X-axis, Y-axis, and Z-axis. You can set the content to be augmented in the correct direction by rotating by 45 degrees by clicking the arrow button. In the case of the same size, you can drag the seekBar to adjust the size to a value between 0 and 100. When the setting values that the registrant satisfies are applied, the user can finish the test by clicking the Done button. When the test is finished, the size and rotation value properties set through the test are applied on the screen where you can perform MarkerConfirm, and you can check. If you want to set the value again, you can change it again by clicking the Test Marker button again.

Therefore, augmented reality can be divided into video, image, and 3D model according to the classification of the content. Each augmented reality content is applied with the size and rotation value set by the marker registrant, so that the same option is applied to all users. . Therefore, even if the image size of the registered markers is different, there is an advantage that an optimized augmented reality content can be experienced through the marker test algorithm of FIG. 20. When the marker is pointed at the camera, if tracking has occurred, a simple vibration on the user's smartphone gives a signal that tracking is occurring. When you press the back button, the augmented reality experience ends.

Meanwhile, the augmented reality server 400 calculates and provides an evaluation value of the marker registered by the advertiser. The marker and business card registration menu for generating augmented reality advertisement content provides a user interface (UI) for receiving and displaying the evaluation value of the marker registered by the advertiser from the augmented reality server 400. The evaluation values of these markers are evaluated by extracting the feature points of the registered markers, grasping the position coordinates, size, and image change direction values at which the feature points are located, and evaluating whether the position coordinates, size, and image change direction values are within the set reference position range. It's one value.

Referring to FIG. 21 in detail, when an image taken by a user in the process of registering a marker is uploaded to an image processing server in order to score marker evaluation, the image processing server prepares for evaluating the image. Proceed. First, the image processing server checks whether the uploaded image has been uploaded normally, and temporarily stores it in the photo folder. After that, the image is uploaded to the memory and feature points are extracted to analyze the image. In order to extract a feature point, a feature point extraction algorithm SIFT or SURF, or various feature point extraction algorithms may be used. After the feature points are extracted, each of the extracted feature points has values for a position coordinate, a size, and a change direction of the image at which the feature points are located in the image. In this algorithm, analysis is performed based on the location coordinates of the extracted feature points.

Analyzing the feature point coordinates is for the purpose of determining whether feature points are concentrated in one direction in an image, and a variance value is used to solve the problem. Since the positional coordinates of the feature points are composed of two dimensions, each with an x-axis and a y-axis, mapping is performed to each axis for convenience of calculating the variance value. After that, the average of the data is obtained to obtain the average position coordinates. In this way, it is possible to check the average of the position coordinates and how far apart each feature point coordinate is through the calculation of the variance value.

Additionally, a process of generalizing the data is required so that the calculated variance values can commonly determine a threshold value in images having different sizes. In this process, min-max normalization is used. Here, the min value is 0 and the max value is the width and height of the image, respectively. When the calculation of the variance value is completed, the variance values for the x-axis and the y-axis are averaged and finally determined as the variance value for one image. The variance value usually has a value between 0.03 and 0.1, but since even an image with a good variance of feature points is rarely more than 0.1, the marker evaluation score is finally calculated by normalizing 0.1 to 5 points. The calculated marker evaluation score is returned to the user, and the user receives the evaluation score in a Float type and displays it on the RatingBar of the smartphone app, and is finished by notifying the exact value of the evaluation score through the Toast message. In addition, if the resolution of the uploaded image is too high, it may place a load on the memory of the server, so the size of the image must be scaled to some extent and implemented stably.

The embodiments in the description of the present invention described above are presented by selecting the most preferable examples to aid the understanding of those skilled in the art from among various possible examples, and the technical idea of the present invention is not necessarily limited or limited only by this embodiment. , Various changes and modifications and other equivalent embodiments are possible within the scope of the technical spirit of the present invention.

100: user terminal
200: advertiser terminal
300: administrator terminal
400: augmented reality server

Claims (12)

A terminal used by an advertiser, comprising: an advertiser terminal installed with an augmented reality advertising platform app for registering advertisement content to be displayed based on a marker and an augmented reality target;
A terminal used by a user, comprising: a user terminal installed with an augmented reality advertising platform app that scans a marker in an area where the user is located and receives and executes an augmented reality targeting the scanned marker from an augmented reality server;
An augmented reality server that generates and stores an augmented reality in which advertisement content is displayed based on the marker registered by the advertiser, and extracts an augmented reality based on the marker requested by the scan of the user terminal and provides it to the user terminal; And
As a terminal used by an administrator managing the augmented reality server, a management UI including user authentication and personal information management UI of the augmented reality server, marker information management UI, content model information management UI, and item purchase information management UI is provided. Includes; a manager terminal that is,
Generating and storing an augmented reality in which advertisement content is displayed based on a marker on which the augmented reality server is registered,
When the selection of the marker and advertisement content is completed by the advertiser, augmented reality that displays advertisement content based on the marker is generated and stored through marker recognition and tracking, visualization test and scale adjustment, and rotation value designation for the user UI. Augmented reality-based advertising platform system through image analysis-based marker production.
The method according to claim 1, wherein the augmented reality advertising platform app,
User authentication menu for user or advertiser authentication, marker and business card registration menu for generating augmented reality advertisement content, marker scan menu for using augmented reality advertisement content, and advertisement items that can be used for augmented reality advertisement content production. An augmented reality-based advertising platform system through image analysis-based marker production including purchase menus.
The method according to claim 2,
An augmented reality-based advertising platform system through image analysis-based marker production, characterized in that when user authentication is successful through the user authentication menu, user location information at the time of authentication is transmitted to and stored in an augmented reality server.
The method according to claim 2, wherein the purchase menu of advertisement items that can be used to produce augmented reality advertisement contents,
Augmented reality-based advertisement through image analysis-based marker production, characterized in that after displaying an advertisement item list, which is a list of pre-registered advertisement items, and then providing a user interface (UI) that enables the purchase of an advertisement item selected by an advertiser. Platform system.
The method according to claim 2, wherein the manager terminal,
An augmented reality-based advertisement platform system through image analysis-based marker production, characterized in that the advertisement item list can be updated and managed through the content model information management UI.
The method according to claim 2, wherein the marker and business card registration menu for generating the augmented reality advertisement content,
A user interface (UI) that allows an advertiser to register a marker including a specific photo or menu in his or her store through the album or camera of the advertiser's terminal, and a map to display the advertiser's location information of the marker to be registered by the advertiser. An augmented reality-based advertising platform system through image analysis-based marker production, characterized in that it provides a user interface (UI) selected from.
The method of claim 6, wherein the marker and business card registration menu for generating the augmented reality advertisement content,
If the marker to be registered is an advertiser's business card, augmentation through image analysis-based marker production characterized by receiving a business card phone number instead of the location information of the marker so that the marker can be identified using an identification number that can recognize the business card. Reality-based advertising platform system.
The method of claim 6, wherein the marker and business card registration menu for generating the augmented reality advertisement content,
After displaying a list of advertisement contents, an augmented reality-based advertisement platform system through image analysis-based marker production, characterized in that it provides a user interface (UI) that enables the purchase of advertisement contents selected by an advertiser.
The method of claim 6,
The augmented reality server calculates and provides an evaluation value of a marker registered by an advertiser,
The marker and business card registration menu for generating the augmented reality advertisement content includes a user interface (UI) for receiving and displaying the evaluation value of the marker registered by the advertiser from the augmented reality server. Augmented reality-based advertising platform system.
The method of claim 9, wherein the evaluation value of the marker,
Characterized in that it is a value obtained by extracting the feature point of the registered marker, grasping the position coordinate, size, and image change direction value at which the feature point is located, and evaluating whether the position coordinate, size, and image change direction value are within a set reference position range. Augmented reality-based advertising platform system through image analysis-based marker production.
delete The method according to claim 1, wherein the scale adjustment and designation of a rotation value for a user UI,
The scale and rotation values are set from the advertiser using the navigation bar and arrow buttons displayed on the advertiser's terminal,
In the case of scale setting, drag the search bar to set the size to any one value within 0~100 steps,
In the case of setting the rotation value, it is divided into X-axis, Y-axis, and Z-axis, and it is based on an augmented reality through image analysis-based marker production characterized in that the direction of the advertisement content is set by rotating by 45 degrees by clicking the arrow button. Advertising platform system.

Images