WO2022102804A1

WO2022102804A1 - Augmented reality-based object control method and provision device

Info

Publication number: WO2022102804A1
Application number: PCT/KR2020/015796
Authority: WO
Inventors: 이동균
Original assignee: 엔티콘 주식회사
Priority date: 2020-11-11
Filing date: 2020-11-11
Publication date: 2022-05-19

Abstract

An augmented reality-based object control provision method according to the present invention may comprise the steps of: (a) executing a camera image-capturing mode of a user terminal having a specific program or application installed thereon, so as to capture an image of an object positioned in front of a user; (b) receiving, from the user terminal, an image captured through the user terminal, and determining whether an image of an object in the received image corresponds to a pre-stored augmented reality (AR) marker image; (c) when the image of the object corresponds to the pre-stored augmented reality marker image, transmitting, to the user terminal, a command to display, in augmented reality, a pre-configured control command identifier on the image received from the user terminal; and (d) when an input of the user of the control command identifier is received from the user terminal, transmitting information on the input of the user to the server by the user terminal, and transmitting a control command corresponding to the input of the user to the object by the server so that the object operates according to the control command.

Description

Augmented reality-based object control method and providing device

The present invention relates to an augmented reality-based content providing method and apparatus, and more particularly, photographing an object through a user terminal, and an image of the photographed object corresponding to an AR (Augmented Reality) marker pre-stored in a server. In this case, the server commands the user terminal to display the control command identifier in augmented reality, and the user terminal inputs the control command to the object through the control command identifier and transmits it to the server, the server sends a control command corresponding to the input of the user terminal It is related to the technology of instructing the thing to operate by sending it to the thing.

Global ICT companies such as Samsung, Google, Microsoft, and HP are also increasing the market size by expanding technology development and investment, expecting VR/AR technology to grow as a future new industry that will follow the smartphone. Although there are differences by research institute in such a bright outlook, the global AR market size is expected to reach about 100 billion dollars in 2022, and the industrial and economic ripple effect is expected to be huge in the future.

The AR market is growing steadily, but on the other hand, there is a limit to mass-producing applications with low business value because it is concentrated on high-end entertainment value, and bubble theory and a sense of crisis about the technology are also emerging because it cannot achieve the results as high as initially expected.

Recently, cases have been reported to improve productivity and increase business value through the convergence of ICT technologies such as the Internet of Things (IoT) and Artificial Intelligence (AI), centering on overseas global companies, but the domestic situation is still Since it is only the beginning stage, a response is required, and research on object control technology using AR is being actively conducted in recent years by reflecting the needs of the times.

The present invention is to solve the problems of the prior art, and when a user shoots an object through a user terminal, and the image of the photographed object corresponds to an AR (Augmented Reality) marker pre-stored in the server, the server When the terminal commands the control command identifier to be displayed in augmented reality, the user terminal inputs the control command to the object through the control command identifier and transmits it to the server, the server transmits the control command corresponding to the input of the user terminal to the object. We want to command things to work.

The augmented reality-based content providing method performed by a server includes the steps of: (a) executing a camera shooting mode of a user terminal in which a specific program or application is installed to shoot an object located in front of the user; (b) receiving an image captured by the user terminal from the user terminal, and determining whether an image of an object in the received image corresponds to a pre-stored AR (Augmented Reality) marker image; (c) when the image of the object corresponds to the pre-stored AR (Augmented Reality) marker image, a command to display a preset control command identifier on the image received from the user terminal in augmented reality, the user terminal sending to; (d) when the user's input for the control command identifier is received from the user terminal, the user terminal transmits information about the user's input to the server, and the server sends the user's input to the thing and sending a control command corresponding to , so that the thing operates according to the control command.

Also, the thing may be an electronic product capable of communicating with the server.

In addition, before step (a), the server providing information about the mission to the user terminal; may include

In addition, the information on the mission is to control the specific electronic product through the AR function of the user terminal, and may include identification information or location information of the specific electronic product and operation information of the electronic product.

In addition, in step (c), the control command identifier may be displayed overlapped with the AR in an area corresponding to the electronic product in the image on the screen of the user terminal.

In addition, when the control command indicates an on/off operation of the electronic product, the control command identifier may be formed in the form of a content play button and displayed on the user terminal.

In addition, the control command identifier may include being set to move together with an image corresponding to the electronic product in the image.

In addition, after step (d), when the electronic product operates according to the control command, the server receives the operation information and determines that the mission is completed; may include.

In addition, when it is determined that the plurality of preset missions have been completed, the method may further include providing compensation information corresponding to the completion of the mission to the user terminal.

In addition, there is provided an apparatus for providing an augmented reality-based object control method, comprising: a memory in which a program related to an augmented reality-based object control method is stored; and a processor executing the program; Including, wherein the processor executes a camera shooting mode of a user terminal in which a specific program or application is installed by executing the program to photograph an object located in front of the user, and displays an image captured through the user terminal to the user Received from the terminal, determine whether an image of an object in the received image corresponds to a pre-stored AR (Augmented Reality) marker image, and if the image for the object corresponds to the pre-stored AR (Augmented Reality) marker image, When a command for displaying a preset control command identifier in augmented reality on the image received from the user terminal is transmitted to the user terminal, and the user's input for the control command identifier is received from the user terminal, the user The terminal may transmit information about the user's input to the server, and the server may transmit a control command corresponding to the user's input to the electronic product to provide a command so that the electronic product operates according to the control command there is.

The present invention captures an object through the camera shooting mode of the user terminal and transmits it to the server, and the server provides a control command identifier that can control the augmented reality content with the user terminal when the received image and the pre-stored AR marker correspond can do. Through this, the user can control the operation of the object through the provided control command identifier.

1 is a diagram illustrating a system configuration for an augmented reality-based object control method according to an embodiment of the present invention.

2 is a diagram showing the configuration of a server according to an embodiment of the present invention.

3 is a flowchart for controlling an electronic product in an augmented reality-based object control method according to an embodiment of the present invention.

4 is an exemplary diagram illustrating an operation of an electronic product corresponding to a control command identifier according to an embodiment of the present invention.

5 is an exemplary diagram of providing augmented reality by recognizing an object in an augmented reality-based object control method according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or to reproduce one or more processors. Accordingly, as an example, '~' indicates components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

The "user terminal" referred to below may be implemented as a computer or portable terminal that can access a server or other terminal through a network. Here, the computer is, for example, a laptop, desktop, laptop, VR HMD (eg, HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.) equipped with a web browser (WEB Browser), etc. may include Here, VR HMD is for PC (eg, HTC VIVE, Oculus Rift, FOVE, Deepon, etc.), for mobile (eg, GearVR, DayDream, Storm Horse, Google Cardboard, etc.) Independently implemented Stand Alone models (eg Deepon, PICO, etc.) are included. A portable terminal is, for example, a wireless communication device that guarantees portability and mobility, and includes not only a smart phone, a tablet PC, a wearable device, but also Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, and ultrasound (Ultrasonic). , infrared, Wi-Fi, Li-Fi, etc. may include various devices equipped with a communication module. In addition, the term "network" refers to a connection structure capable of exchanging information between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet. (WWW: World Wide Web), wired and wireless data networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. include, but are not limited to.

Hereinafter, the term “electronic product” refers to various products including electronic components that can communicate with the server.

Hereinafter, with reference to FIG. 1, the system configuration of the augmented reality-based content providing method according to an embodiment of the present invention will be described in general as follows.

According to an embodiment of the present invention, the system 10 may include a server 100 , a user terminal 200 and an electronic product 300 , and the server 100 includes the user terminal 200 and the server ( 100 ) may be communicatively connected to the electronic product 300 through a wired/wireless network.

According to an embodiment of the present invention, the server 100 receives an image captured through the camera shooting mode of the user terminal 200, and AR (Augmented AR) in which an image of the electronic product 300 in the received image is pre-stored. Reality) it is possible to determine whether it corresponds to the marker image 400 .

When the image of the electronic product 300 corresponds to the pre-stored Augmented Reality (AR) marker image 400 , the server 100 is a preset control command identifier 500 on the image received from the user terminal 200 . transmits a command to be displayed in augmented reality to the user terminal 200, and the user terminal 200 can input a control command including the operation of the target electronic product 300 through the received control command identifier 500. there is.

When the user's input for the control command identifier 500 is received from the user terminal 200, the user terminal 200 transmits information about the user's input to the server 100, and the server 100 is an electronic product. By transmitting a control command corresponding to the user's input to 300 , the electronic product 300 may operate according to the control command.

2 is a diagram showing the configuration of the server 100 according to an embodiment of the present invention.

Referring to FIG. 2 , the server 100 includes a communication module 110 , a memory 120 , and a processor 130 .

In detail, the communication module 110 provides a communication interface necessary to provide a transmission/reception signal between the server 100 and the user terminal 200 and the server 100 and the electronic product 300 in the form of packet data by interworking with the communication network. . Furthermore, the communication module 110 may serve to receive a data request from the terminals and transmit data in response thereto.

Here, the communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

The memory 120 records a program for providing an augmented reality-based object control method. In addition, the processor 130 performs a function of temporarily or permanently storing the processed data. Here, the memory 120 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto.

The processor 130 controls the entire process of providing an augmented reality-based object control method as a kind of central processing unit. Each step performed by the processor 130 will be described later with reference to FIG. 3 .

Here, the processor 130 may include all kinds of devices capable of processing data, such as a processor. Here, the 'processor' may refer to, for example, a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or command included in a program. As an example of the data processing apparatus embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

Hereinafter, an augmented reality-based content providing method according to an embodiment of the present invention will be described in detail with reference to FIG. 3 .

With reference to FIG. 3 , a procedure for controlling electronic products in the augmented reality-based content providing method according to an embodiment of the present invention may be described.

In the augmented reality-based content providing method performed by the server 100 according to an embodiment of the present invention, first, the server 100 may provide information about a mission to the user terminal 200 in advance. At this time, the information about the mission is to control the specific electronic product 300 through the AR function of the user terminal 200 , and information on the identification information or location of the specific electronic product 300 and information on the electronic product 300 . Operational information may be included.

The server 100 may receive an image of an object located in front of the user captured through the camera shooting mode of the user terminal 200 in which a specific program or a specific application is installed. (S710)

In this case, the object located in front of the user may be an electronic product 300 including a TV, DVD, console, AV device, and household appliances that can communicate with the server 100 , and a specific indoor and outdoor space or booklet image may include

The server 100 may determine whether the image of the object in the image received from the user terminal 200 corresponds to the pre-stored AR marker image 400 . (S720)

When the image of the object in the image received from the user terminal 200 corresponds to the AR marker image 400 stored in advance, the server 100 determines a preset control command identifier on the image received from the user terminal 200. You can send commands to be displayed in augmented reality. (S730)

At this time, the control command identifier may be displayed overlapped with AR in an area corresponding to the electronic product 300 in the image on the screen of the user terminal 200, and the control command identifier will be described later with reference to FIG. 4 . .

When the object in the image received from the user terminal 200 is an electronic product 300 capable of communicating with the server 100, the server 100 receives the user's input through the user terminal 200 for the control command identifier. When the user terminal 200 receives the user's input information, it is possible to transmit a control command to the electronic product 300 in response to the received user's input information. (S740)

The electronic product 300 may operate according to the control command transmitted from the server 100 . (S750)

At this time, when the electronic product 300 operates according to the control command transmitted from the server 100 , the server 100 may receive operation information from the electronic product 300 and determine that the mission has been completed. In addition, when it is determined that a plurality of preset missions have been completed, compensation information corresponding to the completion of mission achievement may be provided to the user terminal 200 .

Hereinafter, with reference to FIG. 4 , the control command identifier 500 displayed on the screen of the user terminal 200 may be described.

When the electronic product 300 in front is photographed through the camera shooting mode of the user terminal 200 and transmitted to the server 100 , the server 100 is the user terminal 200 and the control command identifier 500 shown in the drawing. ) may be transmitted to overlap the control command identifier 500 in the form of augmented reality on the image received from the user terminal 200 .

At this time, when the overlapped control command indicates the on/off operation of the electronic product 300 , the control command identifier 500 may be formed in the shape of a content play button and displayed on the user terminal 200 .

as a further example. The shape of the control command identifier 500 may be expressed in various forms such as a circle or a triangle, and may also be expressed in a character form such as ON, OFF, and the like.

In addition, the control command transmitted from the server 100 may indicate an on/off operation or may include volume control, channel change, external input, and additional function control commands.

For example, the case of controlling a door lock through the present invention can be described. When the door lock is photographed using the camera photographing mode of the user terminal, and the captured image and the AR image of the door lock correspond, a control command identifier may be provided to the user terminal. In this case, the control command identifier may be in the form of a number plate for inputting a number, such as a door lock, or may come out in the form of drawing a pattern. When a user input is received through the provided control command identifier, the server may transmit a control command such as opening or closing to the door lock, and the door lock may receive the control command from the server and operate it. In this way, home appliances including smart phones, TVs, and AV devices can be controlled.

Hereinafter, with reference to FIG. 5 , as an embodiment of the present invention, a case in which an object located in front of the user is not an electronic product performing communication but a specific indoor and outdoor space may be described in detail.

The server 100 receives an image of an object located in front of the user photographed through the camera shooting mode of the user terminal 200 in which a specific program or a specific application is installed, and receives an image of an object in the image received from the user terminal 200 It is determined whether the image corresponds to the pre-stored AR marker image 400, but if it does, preset content may be displayed on the screen of the user terminal in augmented reality.

For example, if an AR marker image for a statue is stored in the server in advance, and the user shoots a statue located in front of the user and sends the image of the statue taken to the server, the server will It is determined whether it corresponds to the AR marker image for the stored statue, and if it is, AR content introducing the statue can be displayed on the screen of the user terminal.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module to be executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims

A method of providing object control based on augmented reality, performed by a server,

(a) executing a camera shooting mode of a user terminal in which a specific program or application is installed to photograph an object located in front of the user;

(b) receiving an image captured by the user terminal from the user terminal, and determining whether an image of an object in the received image corresponds to a pre-stored AR (Augmented Reality) marker image;

(c) when the image of the object corresponds to the pre-stored AR (Augmented Reality) marker image, a command to display a preset control command identifier on the image received from the user terminal in augmented reality, the user terminal sending to;

(d) when the user's input for the control command identifier is received from the user terminal, the user terminal transmits information about the user's input to the server, and the server sends the user's input to the thing transmitting a control command corresponding to , so that the thing operates according to the control command; containing,

A method of providing object control based on augmented reality.
The method of claim 1,

The thing is an electronic product that can communicate with the server,

A method of providing object control based on augmented reality.
The method of claim 1,

(a) prior to step,

providing, by the server, information about the mission to the user terminal; which includes,

A method of providing object control based on augmented reality.
4. The method of claim 3,

The information on the mission is to control the specific electronic product through the AR function of the user terminal, and includes identification information or location information of the specific electronic product and operation information of the electronic product,

A method of providing object control based on augmented reality.
The method of claim 1,

In step (c),

The control command identifier is displayed on the screen of the user terminal overlapped with the AR in an area corresponding to the electronic product in the image,

A method of providing object control based on augmented reality.
6. The method of claim 5,

When the control command indicates an on/off operation of the electronic product, the control command identifier is formed in the shape of a content play button and displayed on the user terminal,

A method of providing object control based on augmented reality.
6. The method of claim 5,

wherein the control command identifier is set to move with an image corresponding to an electronic product in the image,

A method of providing object control based on augmented reality.
The method of claim 1,

After step (d),

When the electronic product operates according to the control command, the server receives the operation information and determines that the mission is completed;

A method of providing object control based on augmented reality.
9. The method of claim 8,

When it is determined that the plurality of preset missions have been completed, providing compensation information corresponding to the completion of the mission achievement to the user terminal; further comprising:

A method of providing object control based on augmented reality.
In an apparatus for providing an augmented reality-based object control,

a memory in which a program related to an augmented reality-based object control method is stored; and

a processor executing the program; including,

The processor by the execution of the program

Execute the camera shooting mode of the user terminal in which a specific program or application is installed to photograph an object located in front of the user, receive an image captured through the user terminal from the user terminal, and receive an image of an object in the received image Determines whether or not corresponds to a pre-stored AR (Augmented Reality) marker image, and when the image of the object corresponds to the pre-stored AR (Augmented Reality) marker image, a preset control command identifier on the image received from the user terminal transmits a command to display in augmented reality to the user terminal, but when the user's input for the control command identifier is received from the user terminal, the user terminal transmits information about the user's input to the server and, the server transmits a control command corresponding to the user's input to the electronic product, and provides a command so that the electronic product operates according to the control command.