KR20210112816A - Server, apparatus and method for providing argumented reality service - Google Patents

Abstract

A server, apparatus and method for providing an augmented reality service are disclosed. A server for providing an augmented reality service according to an embodiment includes: a marker generating part for generating a marker for a target object; a content management part for storing one or more target images for the target object and one or more contents authored in each of the one or more target images; and a content providing part for receiving a content request for the target object from a user terminal recognizing the marker, and transmitting the one or more target images and one or more contents to the user terminal.

Description

Server, apparatus and method for providing augmented reality service {SERVER, APPARATUS AND METHOD FOR PROVIDING ARGUMENTED REALITY SERVICE}

The disclosed embodiments relate to augmented reality technology.

In the prior art, if you want to implement augmented reality technology that matches a real object, you need to learn in advance the target object to which the augmented reality is to be applied and put it in an application. Therefore, in the prior art, there is a problem in that in order to expand and apply augmented reality, it is necessary to re-develop and re-learning. That is, it is difficult to expand and apply augmented reality to new objects in the prior art.

In addition, when objects to which augmented reality is applied have similar appearances, it may be difficult to distinguish the objects only with target images obtained by photographing objects having similar appearances. Therefore, the prior art had difficulties in managing augmented reality content.

Specifically, a method for implementing augmented reality in the prior art is as follows.

1) Learning

Create a learning file by learning the target object to implement augmented reality.

2) Content placement

Load the training file from Unity and place the content on the target object.

3) development

Build the application.

4) Target recognition

Run the application to illuminate the learned target object.

5) Show content

Content placed on the target image is tracked and displayed on the target object.

As described above, in the prior art, it is necessary to learn about a professional authoring tool called Unity in order to implement and expand augmented reality. In addition, since the prior art needs to re-build the application, there is a problem in that a general user without developmental knowledge cannot expand and apply augmented reality.

In addition, when a plurality of target images are registered in an application, the prior art has a technical limitation in that it is not possible to enlarge and apply more than a certain number of target objects because the processing speed of the terminal is slowed.

Korean Patent Registration No. 10-1040253 (June 9, 2011 Announcement)

Disclosed embodiments are to provide an augmented reality service providing server, apparatus, and method, for identifying a target object with a marker, recognizing a target image for the target object, and displaying augmented reality content.

An augmented reality service providing server according to an embodiment includes: a marker generating unit generating a marker for a target object; a content management unit for storing one or more target images for the target object and one or more contents authored in each of the one or more target images; and a content providing unit for receiving a content request for the target object from the user terminal recognizing the marker, and transmitting the one or more target images and one or more contents to the user terminal.

The marker generator may generate the marker based on identification information on the target object.

The one or more contents may be authored based on at least one of a user's touch operation and a drag operation on the target image, respectively.

The one or more contents may be authored at a location selected by at least one of the touch operation and the drag operation in the target image, respectively.

An augmented reality service providing method performed by an augmented reality service providing server according to an embodiment includes: generating a marker for a target object; receiving a content request for the target object from a user terminal recognizing the marker; and transmitting one or more target images for the target object and one or more contents authored to each of the one or more target images to the user terminal.

The generating may include generating the marker based on identification information on the target object.

The one or more contents may be authored based on at least one of a user's touch operation and a drag operation on the target image, respectively.

The one or more contents may be authored at a location selected by at least one of the touch operation and the drag operation in the target image, respectively.

An augmented reality service providing apparatus according to an embodiment includes: an object identification unit for recognizing a marker included in an image to identify a target object; Requesting content for the identified target object to an augmented reality service providing server, and receiving one or more target images for the target object and one or more contents authored on each of the one or more target images from the augmented reality service providing server content receiving unit; a target image recognition unit for recognizing at least one of the one or more target images from a currently captured image; and a content display unit displaying one or more contents authored on the recognized target image on the currently captured image.

The object identification unit may identify the target object based on identification information obtained by recognizing the marker.

The content display unit may display one or more contents authored on the recognized target image while tracking the recognized target image in the currently captured image.

A method of providing an augmented reality service according to an embodiment includes: recognizing a marker included in an image; identifying a target object; requesting content for the identified target object to an augmented reality service providing server; receiving, from the augmented reality service providing server, one or more target images of the target object and one or more contents authored in each of the one or more target images; recognizing at least one of the one or more target images from a currently captured image; and displaying one or more contents authored on the recognized target image on the currently captured image.

The identifying may include identifying the target object based on identification information obtained by recognizing the marker.

The displaying may include displaying one or more contents authored on the recognized target image while tracking the recognized target image in the currently captured image.

According to the disclosed embodiments, a target object is identified by recognizing a marker, and content is displayed by recognizing a target image for the target object. Content can be augmented, and augmented reality content management for each target object is easy.

In addition, according to the disclosed embodiments, by receiving one or more target images for the identified target object by recognizing the marker, the user downloads only the target image for the target object desired by the user, and when there are many target images, the user terminal is slowed down. can solve the problem Accordingly, it is possible to enlarge and apply a target object for implementing augmented reality without limitation.

In addition, according to the disclosed embodiments, by storing a target image of a target object and authoring content based on at least one of a user's touch action and a drag action on the target image, general users without developmental knowledge are also enhanced. You can easily develop real content.

1 is a block diagram of an augmented reality service providing system according to an embodiment;
2 is a configuration diagram of an augmented reality service providing server according to an embodiment;
3 is an exemplary view of a marker according to an embodiment;
4 is an exemplary view of a stored target image according to an embodiment;
5 is a diagram for explaining an example of authoring content according to an embodiment;
6 is a block diagram of an apparatus for providing an augmented reality service according to an embodiment;
7 is an exemplary view of a received target image according to an embodiment;
8 is a diagram for explaining an example of displaying content according to an embodiment;
9 is a flowchart of an augmented reality service providing method performed by an augmented reality service providing server according to an embodiment;
10 is a flowchart of a method for providing an augmented reality service performed by an apparatus for providing an augmented reality service according to an embodiment;
11 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, specific embodiments will be described with reference to the drawings. DETAILED DESCRIPTION The following detailed description is provided to provide a comprehensive understanding of the methods, devices, and/or systems described herein. However, this is only an example and is not limited thereto.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions, which may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing the embodiments only, and should in no way be limiting. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. Also, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or combination thereof, and one or more other features than those described. , should not be construed to exclude the presence or possibility of any number, step, action, element, or any part or combination thereof.

1 is a block diagram of an augmented reality service providing system 100 according to an embodiment.

Referring to FIG. 1 , the augmented reality service providing system 100 includes an augmented reality service providing server 110 and a user terminal 130 .

In an embodiment, the augmented reality service providing system 100 may provide an augmented reality service to the user terminal 130 through an augmented reality application installed in the user terminal 130 . In this case, the augmented reality application may be a program for providing an augmented reality service in conjunction with the augmented reality service providing server 110 . For example, the augmented reality application requests a target image and content used to implement augmented reality while communicating with the augmented reality service providing server 110 , and receives the target image and content provided from the augmented reality service providing server 110 . It performs a role of providing to the user terminal 130 and may perform various roles such as content display and user command processing for content.

The augmented reality service providing server 110 may store various types of information used to implement augmented reality, such as identification information on a target object, a marker, a target image, and content.

In this case, the marker is used to identify a target object and augmented reality content corresponding to the target object, and may be configured in the form of a figure having a specific pattern, such as a QR code, a barcode, or the like.

In this case, the target object is an object for which augmented reality is to be implemented.

The target image is an image obtained by photographing the target object.

For example, the augmented reality service providing server 110 may receive various types of information used to implement augmented reality from a user through a website, or may provide the various types of information to the user. In this case, the user accesses a website connected to the augmented reality service providing server 110 through the user terminal 130 or has a display means, a voice processing means, an information processing means, a data communication means, etc. in addition to the user terminal 130 . A website can be accessed through a variety of devices.

In addition, the augmented reality service providing server 110 may provide an augmented reality service in conjunction with the user terminal 130 . For example, when the augmented reality service providing server 110 receives a content request for the target object from the user terminal 130 , the augmented reality service providing server 110 provides the target image for the target object and content authored on the target image to the user terminal 130 . can provide

The user terminal 130 may include various types of devices having a display means, a voice processing means, an information processing means, and a data communication means through a wired/wireless network, such as, for example, a smart phone. In addition, the user terminal 130 may be provided with a photographing means such as a camera.

The user terminal 130 may provide the augmented reality service to the user through the augmented reality application interlocked with the augmented reality service providing server 110 . In addition, the user terminal 130 may output the content provided through the augmented reality service application using an output means such as a display, a speaker, etc., and use an input means such as a touch screen or a keyboard to view the content from the user. command can be entered.

For example, the user terminal 130 recognizes a marker included in a photographed image using a photographing means such as a camera, and provides the augmented reality service providing server 110 with content for a target object corresponding to the recognized marker. you can request In addition, the user terminal 130 receives the target image for the target object and the content authored on the target image from the augmented reality service providing server 110 through the augmented reality application, recognizes the received target image, and recognizes the image currently being photographed. content can be displayed on the

2 is a configuration diagram of an augmented reality service providing server 110 according to an embodiment.

Referring to FIG. 2 , the augmented reality service providing server 110 includes a marker generating unit 111 , a content management unit 113 , and a content providing unit 115 .

In one embodiment, the marker generating unit 111, the content management unit 113, and the content providing unit 115 are implemented using one or more physically separated devices, or one or more processors or a combination of one or more processors and software. may be implemented, and unlike the illustrated example, specific operations may not be clearly distinguished.

The marker generator 111 generates a marker for the target object.

For example, the user may register an object to implement augmented reality as a target object in the augmented reality service providing server 110 . In this case, when the user requests registration of the target object through the website connected to the augmented reality service providing server 110 , the marker generator 111 may generate a marker for the target object.

In an embodiment, as shown in FIG. 3 , the marker generator 111 may generate a marker in the form of a QR code for the target object. However, in FIG. 3 illustrated, the marker is illustrated as a QR code, but the type of the marker is not limited to the QR code and may vary.

In an embodiment, the marker generator 111 may generate a marker based on identification information on the target object.

For example, the marker generator 111 may receive identification information on the target object from the user, and generate a marker based on the received identification information.

However, in the above-described example, it has been described that the identification information for the target object is received from the user, but this is only an exemplary embodiment, and there may be various methods for generating the identification information on the target object. For example, the marker generator 111 may generate identification information on the target object according to a preset algorithm, and generate a marker based on the generated identification information.

Meanwhile, the user may attach the generated marker to the target object, and as will be described later, the marker attached to the target object may be used to identify the target object.

The content management unit 113 stores one or more target images for the target object and one or more contents authored to each of the one or more target images.

In this case, the one or more target images may be images obtained by photographing the target object from different angles. For example, the one or more target images may be images obtained by photographing the front, rear, left, and right surfaces of the target object. Accordingly, all sides of the target object may be targets for implementing augmented reality.

In an embodiment, the content management unit 113 may match one or more target images for a target object and one or more contents authored to each of the one or more target images with identification information and store them.

For example, as shown in FIG. 4 , a user may register one or more target images 410 for a target object through a website connected to the augmented reality service providing server 110 . In this case, the content manager 113 may match and store one or more target images of the target object input by the user with identification information of the target object.

In addition, the user may author one or more contents for each of one or more target images through a website connected to the augmented reality service providing server 110 .

For example, as shown in FIG. 5 , the augmented reality service providing server 110 may provide a user interface 500 for content authoring to a user through a website. In this case, the user interface 500 may include a target image list 510 , a target image 520 for creating content, a content list 530 , and the like. In this case, the content may include various types of content, such as text content 531 and image content 533 .

In an embodiment, the one or more contents may be authored based on at least one of a user's touch operation and a drag operation on the target image, respectively. Specifically, the user may select a target image for authoring content among one or more target images. Thereafter, the user may freely arrange the content at a desired position in the target image through a touch operation, a drag operation, or the like using an input means such as a touch screen or a mouse. In this case, each of the one or more contents may be authored at a location selected by at least one of a user's touch operation and a drag operation within the target image. Thereafter, the content management unit 113 may store content authoring information such as content, form, and location of the content authored in the target image.

The content providing unit 115 receives a content request for a target object from the user terminal 130 recognizing the marker, and provides one or more target images for the target object and one or more contents authored on each of the one or more target images to the user terminal. (130).

For example, when receiving a content request for a target object from the user terminal 130 , the content providing unit 115 authorizes one or more target images and one or more target images stored by matching identification information on the target object. One or more contents can be loaded. Thereafter, the content providing unit 115 may transmit one or more target images of the target object and one or more contents authored to each of the one or more target images to the user terminal 130 .

6 is a configuration diagram of an apparatus 200 for providing an augmented reality service according to an embodiment.

The augmented reality service providing apparatus 600 illustrated in FIG. 6 may be implemented, for example, in a configuration included in the user terminal 130 illustrated in FIG. 1 .

Referring to FIG. 6 , the augmented reality service providing apparatus 600 includes an object identification unit 610 , a content receiving unit 630 , a target image recognition unit 650 , and a content display unit 670 .

In one embodiment, the object identification unit 610, the content receiving unit 630, the target image recognition unit 650, and the content display unit 670 are implemented using one or more physically separated devices, or one or more processors or one It may be implemented by a combination of the above processor and software, and may not be clearly distinguished in specific operation unlike the illustrated example.

The object identification unit 610 identifies a target object by recognizing a marker included in the image.

For example, when the user scans a marker attached to the target object using the photographing means, the object identification unit 610 may identify the target object by recognizing the marker included in the image acquired by the photographing means. .

In an embodiment, the object identification unit 610 may identify a target object based on identification information obtained by recognizing a marker.

The content receiving unit 630 requests the content for the target object identified by the augmented reality service providing server 110, and from the augmented reality service providing server 110 to one or more target images and one or more target images for the target object, respectively. Receive one or more authored content.

For example, as shown in FIG. 7 , when a target object is identified by scanning a marker attached to the target object, the content receiving unit 630 receives one or more targets for the target object from the augmented reality service providing server 110 . One or more content authored on each of the image and the one or more target images may be downloaded. In this case, the user may check the downloaded target image through the display provided in the user terminal 130 . In addition, the downloaded target image may be immediately learned and stored in the augmented reality service providing apparatus 600 in the form of a learning file required for target image recognition and tracking, which will be described later.

The target image recognition unit 650 recognizes at least one of one or more target images in the current captured image.

For example, the target image recognition unit 650 may recognize the target image using a predefined image recognition algorithm, a deep learning model for image recognition, or the like.

The object display unit 670 displays one or more contents authored on the recognized target image on the currently captured image.

For example, as shown in FIG. 8 , when a specific target image 810 is recognized from the currently captured image 800 , the object display unit 670 displays one or more contents authored on the recognized target image 810 currently. The captured image 800 may be overlaid and displayed. The object display unit 670 may overlay and display the content on the currently captured image 800 through an output means such as a monitor or a display device. In this case, the content 531 in the form of text, the content 533 in the form of an image, etc. written on the target image 810 may be displayed on the currently captured image 800 .

In an embodiment, the object display unit 670 may display one or more contents authored on the recognized target image while tracking the recognized target image in the current captured image.

In this case, the target image tracking may be performed using various predefined image tracking algorithms such as feature point tracking, for example.

9 is a flowchart of an augmented reality service providing method performed by an augmented reality service providing server according to an embodiment.

The method illustrated in FIG. 9 may be performed, for example, by the augmented reality service providing server 110 illustrated in FIG. 2 .

Referring to FIG. 9 , the augmented reality service providing server 110 generates a marker for a target object ( 910 ).

In this case, the augmented reality service providing server 110 may generate a marker based on identification information on the target object.

Thereafter, the augmented reality service providing server 110 receives a content request for the target object from the user terminal 130 recognizing the marker ( 920 ).

Thereafter, the augmented reality service providing server 110 transmits one or more target images of the target object and one or more contents authored on each of the one or more target images to the user terminal 130 ( 930 ).

10 is a flowchart of a method for providing an augmented reality service performed by an apparatus for providing an augmented reality service according to an exemplary embodiment.

The method illustrated in FIG. 10 may be performed, for example, by the augmented reality service providing apparatus 600 illustrated in FIG. 6 .

Referring to FIG. 10 , the augmented reality service providing apparatus 600 recognizes a marker included in an image ( 1010 ).

Thereafter, the augmented reality service providing apparatus 600 identifies the target object ( 1020 ).

In this case, the augmented reality service providing apparatus 600 may identify the target object based on identification information obtained by recognizing the marker.

Thereafter, the augmented reality service providing apparatus 600 requests the augmented reality service providing server 110 for content on the identified target object based on the marker ( 1030 ).

Thereafter, the augmented reality service providing apparatus 600 receives, from the augmented reality service providing server 110 , one or more target images of a target object and one or more contents authored on each of the one or more target images ( 1040 ).

Thereafter, the augmented reality service providing apparatus 600 recognizes at least one of one or more target images from the currently captured image ( S1050 ).

Thereafter, the augmented reality service providing apparatus 600 displays one or more contents authored on the recognized target image on the currently captured image ( 1060 ).

In this case, the augmented reality service providing apparatus 600 may display one or more contents authored on the recognized target image while tracking the recognized target image in the current captured image.

On the other hand, in the flowcharts shown in FIGS. 9 and 10, the method is described by dividing the method into a plurality of steps, but at least some of the steps are performed in a different order, performed together in combination with other steps, omitted, or divided into detailed steps. It may be performed, or one or more steps not shown may be added and performed.

11 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, the computing device 12 may be, for example, one or more components included in the augmented reality service providing server 110 shown in FIG. 2 and the augmented reality service providing device 600 shown in FIG. 6 . have.

Computing device 12 includes at least one processor 14 , computer readable storage medium 16 , and communication bus 18 . The processor 14 may cause the computing device 12 to operate in accordance with the exemplary embodiments discussed above. For example, the processor 14 may execute one or more programs stored in the computer-readable storage medium 16 . The one or more programs may include one or more computer-executable instructions that, when executed by the processor 14, configure the computing device 12 to perform operations in accordance with the exemplary embodiment. can be

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer readable storage medium 16 includes a set of instructions executable by the processor 14 . In one embodiment, computer-readable storage medium 16 includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, other forms of storage medium accessed by computing device 12 and capable of storing desired information, or a suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12 , including processor 14 and computer readable storage medium 16 .

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . The input/output interface 22 and the network communication interface 26 are coupled to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output device 24 may include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or imaging devices. input devices, and/or output devices such as display devices, printers, speakers and/or network cards. The exemplary input/output device 24 may be included in the computing device 12 as a component constituting the computing device 12 , and may be connected to the computing device 12 as a separate device distinct from the computing device 12 . may be

In the above, the technical features have been described focusing on the embodiments. However, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view, and the scope of the rights is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto are interpreted as included in the scope of rights. it should be

10: Computing Environment
12: computing device
14: Processor
16: computer readable storage medium
18: communication bus
20: Program
22: input/output interface
24: input/output device
26: network communication interface
100: augmented reality service providing system
110: augmented reality service providing server
130: user terminal
111: marker generating unit
113: content management department
115: content provider
600: augmented reality service providing device
610: object identification unit
630: content receiving unit
650: target image recognition unit
670: content display unit

Claims (14)

a marker generator generating a marker for the target object;
a content management unit for storing one or more target images for the target object and one or more contents authored in each of the one or more target images; and
Augmented reality service providing server comprising a content providing unit for receiving a content request for the target object from the user terminal recognizing the marker, and transmitting the one or more target images and one or more contents to the user terminal.
The method according to claim 1,
The marker generating unit is an augmented reality service providing server that generates the marker based on identification information on the target object.
The method according to claim 1,
The one or more contents are, respectively, an augmented reality service providing server that is authored based on at least one of a user's touch operation and a drag operation on the target image.
4. The method according to claim 3,
The one or more contents are each authored at a location selected by at least one of the touch action and the drag action in the target image, an augmented reality service providing server.
In the augmented reality service providing method performed by the augmented reality service providing server,
generating a marker for the target object;
receiving a content request for the target object from a user terminal recognizing the marker; and
and transmitting one or more target images for the target object and one or more contents authored on each of the one or more target images to the user terminal.
6. The method of claim 5,
The generating may include generating the marker based on identification information on the target object.
6. The method of claim 5,
The one or more contents are each authored based on at least one of a user's touch operation and a drag operation on the target image.
8. The method of claim 7,
The one or more contents are each authored at a location selected by at least one of the touch operation and the drag operation in the target image.
an object identification unit for recognizing a marker included in an image to identify a target object;
Requesting content for the identified target object to an augmented reality service providing server, and receiving one or more target images for the target object and one or more contents authored on each of the one or more target images from the augmented reality service providing server content receiving unit;
a target image recognition unit for recognizing at least one of the one or more target images from a currently captured image; and
and a content display unit for displaying one or more contents authored on the recognized target image on the currently captured image.
10. The method of claim 9,
The object identification unit may be configured to identify the target object based on identification information obtained by recognizing the marker.
10. The method of claim 9,
The content display unit, while tracking the recognized target image in the currently captured image, the augmented reality service providing apparatus for displaying one or more contents authored on the recognized target image.
recognizing a marker included in the image;
identifying a target object;
requesting content for the identified target object to an augmented reality service providing server;
receiving, from the augmented reality service providing server, one or more target images of the target object and one or more contents authored in each of the one or more target images;
recognizing at least one of the one or more target images from a currently captured image; and
and displaying one or more contents authored on the recognized target image on the currently captured image.
13. The method of claim 12,
The identifying may include identifying the target object based on identification information obtained by recognizing the marker.
13. The method of claim 12,
The displaying may include displaying one or more contents authored on the recognized target image while tracking the recognized target image in the currently captured image.

Images