KR102338984B1 - System for providing 3D model augmented reality service using AI and method thereof - Google Patents

Abstract

The present invention relates to a system and method for providing a 3D model augmented reality service using artificial intelligence. a content providing server that provides, generates a scenario operated by combining with a 3D model according to the 3D model design data, and produces and provides augmented reality (AR) content according to the generated scenario; and using an artificial intelligence (AI)-based marker recognition algorithm in a 3D A model to which an AR marker is attached for reproducing the AR content, and recognizes the AR marker according to the AR content synchronized with the AR marker. It may be a system including an AR service providing terminal that provides an AR image.

Description

A system for providing 3D model augmented reality service using artificial intelligence and a method thereof { System for providing 3D model augmented reality service using AI and method thereof }

The present invention relates to a 3D model augmented reality service providing system and method using artificial intelligence that can provide AR images by combining 3D models and AR contents.

In accordance with the recent global trend in which product design and appearance are important, model production for style verification plays a major role in various fields such as home appliances, automobiles, and architecture. It should be able to express itself well, and the time and cost required for production should be small.

In this case, the model is an object made by imitating the real thing, meaning a wide range of models that are generally used, and can be applied to various fields such as industrial models, human body models, topographic models, and complex models.

In the field of virtual reality technology, a technology that gives a three-dimensional effect to an image using special equipment such as three-dimensional glasses has been developed, but this technology is cumbersome to use, does not satisfy the three-dimensional effect, and requires expensive equipment. In addition, in the field of rapid prototyping, a technology for easily manufacturing models by stacking various 3D models has been developed. Since it requires subsequent work such as painting, additional cost and time are required.

In addition, in the field of computer graphics, shading and texture mapping technologies are used to create realistic images from product shape information and material information. It has advantages and is widely used for design verification in the fields of home appliances, automobiles, architecture, etc., but has a disadvantage in that it lacks a three-dimensional effect.

Therefore, 3D models being produced in various fields, such as museums, exhibition halls, medical sites, industrial sites, and educational sites, apply augmented reality technology to show realistic visual effects, and AR contents combined with 3D models on the terminal A technology that can increase the recognition rate of AR markers should be developed so that they can be reproduced without interruption.

In order to solve the above problems, the present invention recognizes an AR marker attached to a 3D model produced according to a user's request according to an embodiment of the present invention through an artificial intelligence-based marker recognition algorithm, and combines it with the 3D model. The purpose is to make it possible to play AR contents that are in operation.

However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

As a technical means for achieving the above technical task, the 3D model augmented reality service providing system using artificial intelligence according to an embodiment of the present invention receives model production request information for a 3D model from a user terminal, the model Based on the production request information, 3D model design data is provided, a scenario that is operated by combining with a 3D model according to the 3D model design data is generated, and Augmented Reality (AR) content is produced according to the generated scenario a content providing server provided by and using an artificial intelligence (AI) based marker recognition algorithm in a 3D model to which an AR marker is attached to reproduce the AR content, and recognizes the AR marker according to the AR content synchronized with the AR marker. An AR service providing terminal that provides an image, wherein the artificial intelligence-based marker recognition algorithm extracts marker information through raw data obtained from a sensor or camera, and the extracted marker through a pre-learned object detection model Recognizing AR markers in information.

3D model augmented reality service providing system using artificial intelligence, when 3D model design data including AR marker location information is received from the content providing server, creates a 3D model based on the 3D model design data, and the AR The apparatus may further include a model making device for attaching the AR marker to the manufactured 3D model according to the marker position information.

The content providing server transmits one or more preliminary scenarios to the user terminal, and when the edit request data for the preliminary scenario is received from the user terminal, reflects the edit request data to determine the scenario.

The AR service providing terminal generates a candidate area for object recognition through image analysis by performing the artificial intelligence-based marker recognition algorithm, extracting one or more objects from the generated candidate area, and then using the object detection model an AI engine for classifying and detecting an AR marker among the one or more extracted objects; and an AR engine for displaying an AR image by combining the AR content with a 3D model by the recognized AR marker.

In this case, the AR engine may include: an editing module for authoring a virtual scene and a virtual object according to the AR content; a shape generating module that creates a shape of the virtual object; a rendering module for generating a scene displayed on the screen based on the lighting information and the camera information; and a simulation module that examines a collision between the virtual object and the real environment model and generates a physical reaction between the virtual object and the real environment model.

Meanwhile, the AI engine may use a CNN model for object detection of at least one of R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, SSD, and YOLO.

The content providing server encrypts data based on authentication of the AR marker, distribution of a secret key for the authenticated AR marker, and the distributed secret key to prevent exposure and falsification of AR content synchronized with the AR marker A lightweight cryptographic algorithm can be implemented.

On the other hand, the 3D model augmented reality service providing method using artificial intelligence according to another embodiment of the present invention utilizes artificial intelligence performed by a content providing server that provides augmented reality (AR) content. A method for providing a 3D model augmented reality service, comprising: a) receiving an access request from a content providing server generated when a user terminal executes a predetermined application or accesses a predetermined website, and provides a 3D model service page to the user terminal providing; b) receiving model production request information including a target object and purpose of a 3D model through the 3D model service provision page from the user terminal, and providing 3D model design data based on the model production request information; c) generating a scenario to be operated by combining with a 3D model according to the 3D model design data, and registering Augmented Reality (AR) content according to the generated scenario to an AR marker; and d) when AR content request information according to the AR marker recognition is received, providing AR content registered to the AR marker.

In step d), authentication of the AR marker, distribution of a secret key to the authenticated AR marker, and data encryption based on the distributed secret key to prevent exposure and tampering of AR content synchronized with the AR marker It implements a lightweight cryptographic algorithm that

In step d), the AR marker attached to the 3D model is recognized by performing the artificial intelligence-based marker recognition algorithm, and the AR content is combined with the 3D model by the recognized AR marker to display the AR image. The AR content request information is received from the AR service providing terminal.

In step c), one or more preliminary scenarios are transmitted to the user terminal, and when edit request data for the preliminary scenario is received from the user terminal, a scenario may be determined by reflecting the edit request data.

According to the problem solving means of the present invention described above, it is possible to provide an augmented reality-based 3D model in various fields such as exhibition halls, museums, industry, education, and medical care, and provides a service in the form of a realistic experience by combining with the 3D model can do.

For example, in the case of a museum, rather than viewing a model in a static state, visually realistic information can be provided using an augmented reality-based 3D model, and guide information about the 3D model itself can be provided.

In addition, the present invention can implement a lightweight encryption system in a limited environment such as a smart device by using a lightweight encryption algorithm during AR marker authentication, and can prevent exposure and tampering of AR content synchronized with the AR marker, and Through an intelligence-based marker recognition algorithm, AR markers of various shapes, sizes, and colors can be accurately recognized, increasing the marker recognition rate, and continuously augmenting AR content according to the position and angle of the recognized AR marker. have.

1 is a view for explaining the configuration of a 3D model augmented reality service providing system utilizing artificial intelligence according to an embodiment of the present invention.
2 is a diagram for explaining the configuration of an AR service providing terminal according to an embodiment of the present invention.
3 is a flowchart illustrating a method of providing a 3D model augmented reality service using artificial intelligence according to an embodiment of the present invention.
4 is an exemplary diagram illustrating AR content reproduction by an AR service providing terminal according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In the present specification, a 'terminal' may be a wireless communication device with guaranteed portability and mobility, for example, any type of handheld-based wireless communication device such as a smartphone, tablet PC, or notebook computer. In addition, the 'terminal' may be a wired communication device such as a PC that can connect to another terminal or server through a network. In addition, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW). : World Wide Web), wired and wireless data networks, telephone networks, and wired and wireless television networks.

Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, and the like are included, but are not limited thereto.

The following examples are detailed descriptions to help the understanding of the present invention, and do not limit the scope of the present invention. Accordingly, an invention of the same scope performing the same function as the present invention will also fall within the scope of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining the configuration of a 3D model augmented reality service providing system utilizing artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 1 , the 3D model augmented reality service providing system using artificial intelligence includes, but is not limited to, a content providing server 100 , a model making device 200 , and an AR service providing terminal 300 . '

The content providing server 100 may be a computer main body for a server in a general sense, or may be implemented as various types of devices capable of performing a server role. Specifically, the content providing server 100 may be implemented in a computing device including a communication module (not shown), a memory (not shown), a processor (not shown), and a database (not shown), and a mobile phone, TV, or PDA. , tablet PC, PC, notebook PC, and other user terminals may be implemented.

When the content providing server 100 receives the model making request information for the 3D model from the user terminal 10, it provides 3D model design data according to the model making request information, and combines it with the 3D model according to the 3D model design data. It creates an operating scenario, and creates and provides augmented reality (AR) content according to the generated scenario.

The content providing server 100 is a lightweight encryption method that encrypts data based on authentication of AR markers, distribution of secret keys to the authenticated AR markers, and distribution of secret keys to prevent exposure and falsification of AR content synchronized with AR markers run the algorithm. The lightweight encryption algorithm is an encryption technology designed to be implemented in limited environments such as smart devices, and includes symmetric key encryption algorithms such as HIGHT (HIGh security and liht weigHT), LEA (Lightweight Encryption), and hash function LSH (Lightweight Secure Hash). have.

When the 3D model design data including the AR marker location information is received from the content providing server 100, the model making device 200 produces a 3D model based on the 3D model design data, and the 3D model according to the AR marker location information is received. Attach an AR marker to the

The AR service providing terminal 300 recognizes an AR marker using an artificial intelligence (AI) based marker recognition algorithm in a 3D model to which the AR marker is attached, and provides an AR image according to the AR content synchronized with the AR marker. In this case, the artificial intelligence-based marker recognition algorithm may extract marker information through raw data obtained from a sensor or camera, and recognize the AR marker through a pre-learned object detection model based on the extracted marker information.

2 is a diagram for explaining the configuration of an AR service providing terminal according to an embodiment of the present invention.

Referring to FIG. 2 , the AR service providing terminal 300 includes an AI engine 310 and an AR engine 320 , but further includes a communication module (not shown), a memory (not shown), a processor (not shown), and the like. may include

The AI engine 310 simultaneously extracts the type of object and the marker information, which is 3D geometric information, from data obtained from a sensor such as GPS or IMU (Inertial Measurement Unit) or from a camera image obtained from a camera, and according to the marker information, a real environment object Perform 3D transformation or addition of 3D virtual objects to the 3D model. To this end, the AI engine 310 includes modules such as an object analysis module, an object and background analysis module, a plane/depth analysis module, a 3D position estimation module, and a rendering module.

The AI engine 310 performs an artificial intelligence-based marker recognition algorithm to generate a candidate region for object recognition through image analysis, extracts one or more objects from the generated candidate region, and then a pre-trained object detection model AR markers are recognized among one or more objects extracted through The object detection model uses at least one CNN model for object detection among R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, SSD, and YOLO.

In particular, R-CNN generates a candidate region for AR marker recognition, and detects the position of an object in an image by learning the CNN based on the generated candidate region. That is, candidate regions are generated from the input camera image using a selective search algorithm, each generated candidate region is converted to the same size, features are extracted through CNN, and objects in the candidate region are identified using the extracted features. Classify using SVM. SSD (Single Shot Detector) recognizes objects using RPN (Resion Proposal Network) for generating candidate regions, that is, using feature maps of various sizes without training CNN separately. As the product layer progresses, the size decreases. The SSD uses all the feature maps extracted in this process in the inference process to recognize the object.

As such, the AI-based marker recognition algorithm can accurately recognize the shape, size, and color of various AR markers using an object detection model learned using a large-capacity learning dataset, thereby increasing the marker recognition rate. AR content can be continuously augmented according to the position and angle of the marker.

The AR engine 320 combines the AR content with the real environment model by the AR marker and displays it, and includes an editing module 321 , a shape generating module 322 , a rendering module 323 , and a simulation module 324 . However, the present invention is not limited thereto.

The editing module 321 authores a virtual scene and a virtual object according to the AR content, the shape generating module 322 creates the shape of the virtual object, and the rendering module 323 is displayed on the screen based on the lighting information and the camera information Creating a scene, the simulation module 324 examines the collision between the virtual object and the real environment model, and generates an appropriate physical reaction between the virtual object and the real environment model.

The AR service providing terminal 300 may be any type of handheld-based wireless communication device, such as a smartphone, tablet PC, or notebook equipped with the AI engine 310 and the AR engine 320 .

When the user terminal 10 that has transmitted the model making request information for the 3D model is equipped with the AI engine 310 and the AR engine 320 , the user terminal 10 becomes the AR service providing terminal 300 . can

3 is a flowchart illustrating a method of providing a 3D model augmented reality service using artificial intelligence according to an embodiment of the present invention, and FIG. 4 is an AR content reproduction by an AR service providing terminal according to an embodiment of the present invention. It is an example diagram to explain.

3 and 4, the 3D model augmented reality service providing method using artificial intelligence, the content providing server 100 as the user terminal 10 executes a predetermined application or accesses a predetermined website When an access request is received from the generated content providing server 100 (S1), a 3D model service provision page is provided to the user terminal 10 (S2).

The user terminal 10 transmits the model making request information including the target object and purpose of the 3D model to the content providing server 100 through the 3D model service providing page (S3), and the content providing server 100 sends the model making request After generating 3D model design data based on the information, the 3D model design data is transmitted at the request of the user terminal 10 (S4).

The content providing server 100 creates a scenario that is operated by combining with the 3D model according to the 3D model design data, and creates AR content according to the generated scenario and registers it in the AR marker (S5, S6). In this case, the content providing server 100 includes information on the location of the AR marker to which the AR marker is to be attached in the 3D model design data according to the scenario. Accordingly, the model making apparatus 200 attaches the AR marker to the 3D model based on the AR marker position information after manufacturing the 3D model according to the 3D model design data including the AR marker position information.

Meanwhile, the content providing server 100 transmits one or more preliminary scenarios to the user terminal 10 , and when editing request data for the preliminary scenario is received from the user terminal 10 , the scenario is determined by reflecting the editing request data. In this way, the content providing server 100 may create an optimal scenario by reflecting the user's opinion through collaboration with the user terminal 10 . In particular, in the case of a 3D model in the industrial field, the content providing server 100 may generate a scenario by reflecting professional user opinions such as process flow, machine operation process, and the like, thereby providing accurate visual information.

When AR content request information according to AR marker recognition is received from the AR service providing terminal 300 (S7, S8), the content providing server 100 provides AR content synchronized with the AR marker (S9), and provides AR service The terminal 300 outputs an AR image according to AR content by combining it with a 3D model as shown in FIG. 4 ( S10 ).

As described above, the present invention can provide an augmented reality-based 3D model in various fields such as exhibition halls, museums, industry, education, and medical care, and can provide a service in the form of a realistic experience by combining with the 3D model.

The embodiments of the present invention described above may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Such recording media includes computer-readable media, and computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Computer readable media also includes computer storage media, which include volatile and nonvolatile embodied in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. , including both removable and non-removable media.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: user terminal 100: content providing server
200: model making device 300: AR service providing terminal

Claims (12)

When the model production request information for the 3D model is received from the user terminal, 3D model design data is provided based on the model production request information, and a scenario that is operated in combination with the 3D model according to the 3D model design data is generated, a content providing server that creates and provides augmented reality (AR) content according to the generated scenario; and
Recognizes the AR marker using an artificial intelligence (AI) based marker recognition algorithm in a 3D A model with an AR marker attached to play the AR content, and AR according to the AR content synchronized with the AR marker Including an AR service providing terminal that provides an image,
The artificial intelligence-based marker recognition algorithm extracts marker information through raw data obtained from a sensor or camera, and recognizes an AR marker from the extracted marker information through a pre-learned object detection model,
The AR service providing terminal,
The artificial intelligence-based marker recognition algorithm is performed to generate a candidate region for object recognition through image analysis, extract one or more objects from the generated candidate region, and then extract one or more objects through the object detection model AI engine that classifies and detects AR markers from among; and
and an AR engine for displaying an AR image by combining the AR content with a 3D model by the recognized AR marker,
The AR engine is
an editing module for authoring virtual scenes and virtual objects according to the AR contents;
a shape generating module that creates a shape of the virtual object;
a rendering module for generating a scene displayed on the screen based on the lighting information and the camera information; and
3D model augmented reality service providing system using artificial intelligence including a simulation module that inspects a collision between the virtual object and the real environment model and generates a physical reaction between the virtual object and the real environment model. The method of claim 1,
When 3D model design data including AR marker location information is received from the content providing server, a 3D model is produced based on the 3D model design data, and the AR marker is placed on the manufactured 3D model according to the AR marker location information. 3D model augmented reality service providing system using artificial intelligence, which will further include a model making device for attaching a. The method of claim 1,
The content providing server,
3D model augmented reality using artificial intelligence to transmit one or more preliminary scenarios to the user terminal, and to determine a scenario by reflecting the editing request data when editing request data for the preliminary scenario is received from the user terminal service delivery system. delete delete The method of claim 1,
The AI engine is
R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, SSD, YOLO, which uses at least one CNN model for object detection, a 3D model augmented reality service providing system using artificial intelligence. The method of claim 1,
The content providing server,
In order to prevent exposure and falsification of AR content synchronized with the AR marker, authentication of the AR marker, distribution of a secret key to the authenticated AR marker, and a lightweight encryption algorithm for encrypting data based on the distributed secret key are performed 3D model augmented reality service providing system using artificial intelligence. In the method of providing a 3D model augmented reality service using artificial intelligence performed by a content providing server that provides augmented reality (AR) content,
a) receiving an access request from a content providing server generated when the user terminal executes a predetermined application or accesses a predetermined website, and provides a 3D model service provision page to the user terminal;
b) receiving model production request information including a target object and purpose of a 3D model through the 3D model service provision page from the user terminal, and providing 3D model design data based on the model production request information;
c) generating a scenario to be operated by combining with a 3D model according to the 3D model design data, and registering Augmented Reality (AR) content according to the generated scenario to an AR marker; and
d) when AR content request information according to the recognition of the AR marker is received, providing AR content registered to the AR marker;
Step d) is,
From the AR service providing terminal that recognizes the AR marker attached to the 3D model by performing the artificial intelligence-based marker recognition algorithm, and displays the AR image by combining the AR content with the 3D model by the recognized AR marker. Receive AR content request information,
The AR service providing terminal,
The artificial intelligence-based marker recognition algorithm is performed to generate a candidate region for object recognition through image analysis, and after extracting one or more objects from the generated candidate region, from among the one or more extracted objects through an object detection model AI engine that classifies and detects AR markers; and
and an AR engine for displaying an AR image by combining the AR content with a 3D model by the recognized AR marker,
The AR engine is
an editing module for authoring virtual scenes and virtual objects according to the AR contents;
a shape generating module that creates a shape of the virtual object;
a rendering module for generating a scene displayed on the screen based on the lighting information and the camera information; and
A method of providing a 3D model augmented reality service using artificial intelligence, comprising a simulation module that inspects a collision between the virtual object and the real environment model and generates a physical reaction between the virtual object and the real environment model. 9. The method of claim 8,
Step d) is,
In order to prevent exposure and falsification of AR content synchronized with the AR marker, authentication of the AR marker, distribution of a secret key to the authenticated AR marker, and a lightweight encryption algorithm for encrypting data based on the distributed secret key are performed A method of providing a 3D model augmented reality service using artificial intelligence. delete 9. The method of claim 8,
Step c) is,
3D model augmented reality using artificial intelligence to transmit one or more preliminary scenarios to the user terminal, and to determine a scenario by reflecting the editing request data when editing request data for the preliminary scenario is received from the user terminal HOW TO PROVIDE SERVICES. A computer-readable recording medium in which a program for performing the method of providing a 3D model augmented reality service using artificial intelligence according to claim 8 is recorded.

Images