WO2020213783A1

WO2020213783A1 - System and method for providing user interface of virtual interactive content, and recording medium having computer program stored therein for same

Info

Publication number: WO2020213783A1
Application number: PCT/KR2019/006028
Authority: WO
Inventors: 고종필
Original assignee: 주식회사 지티온
Priority date: 2019-04-17
Filing date: 2019-05-20
Publication date: 2020-10-22
Also published as: KR102054148B1; KR102275702B1; WO2020213784A1; KR102041279B1; KR20200122202A

Abstract

The present invention relates to a technology for providing a user interface for a virtual interactive content by utilizing a wall surface or a floor surface as a screen. More specifically, the present invention relates to a technology for providing a user interface in order to play a virtual interactive content projected on a wall surface or a floor surface, using a virtual mouse object such as a soccer ball. A moving object is identified from an image obtained by a digital camera by capturing a situation in which a content is played, and a moving path of the object is tracked to generate an event corresponding to a click of a mouse at the moment when the object reaches the wall surface. For clear identification of an object, a characteristic pattern of the object may be learned in advance through machine learning.

Description

A system and method for providing a user interface for virtual interactive contents, and a recording medium storing a computer program therefor

The present invention relates to a technology for providing a user interface for virtual interactive content using a wall or a floor as a screen. More specifically, the present invention relates to a technology for providing a user interface for playing virtual interactive content projected on a wall or a floor using a virtual mouse object such as a soccer ball. A moving object is identified in an image captured with a digital camera of the content being played, and the movement of the object is tracked to generate an event corresponding to a mouse click when the object touches the wall. In order to clearly identify the object, the characteristic pattern of the object can be learned in advance through machine learning.

The technology of projecting interactive contents on a large screen such as a wall and executing interactive contents using a virtual mouse such as a ball has recently been in the spotlight. In particular, in the field of sports education, interactive indoor sports contents that can be enjoyed indoors regardless of environmental conditions such as outdoor temperature, fine dust concentration, rainfall, and snowfall are gradually being introduced.

For such interactive content, it is essential to recognize an object such as a ball that plays the role of a virtual mouse, track the movement of the object, and find the coordinates of the moment when the object is touched.

Machine vision technology has been proposed as a way to implement these functions. Machine vision technology refers to a combination of hardware and software that provides operating instructions for devices that perform functions of capturing and processing images, and has been used primarily as a technology to manage the production quality of products in various industries.

One of the machine vision technologies is a 3 Dimensional Depth camera technology that uses an infrared (IR) camera. The infrared camera includes at least one infrared light irradiation module and at least one light sensor module, and for all pixels of a photographed image, a camera using a lag or a phase shift of the modulated optical signal ( 10) It uses the so-called ToF method (Time-Of-Flight measurement) to measure the distance between the moving object.

However, since interactive content using an infrared camera can accurately track an object in a dark environment under a certain level of illumination, there is a hassle of additionally awning in the room where the content is executed, and when exposed to daylight, the recognition rate of the object rapidly decreases. there is a problem.

A technique for identifying and tracking an object corresponding to a virtual mouse through digital processing of a play image of a content captured using a general digital camera has also been proposed.

However, even if the object identification success rate is not satisfactory and even if it is set to recognize a specific object, the two objects are properly identified when the first object that needs to be identified in the same image and the second object that is partially similar to the first object but should not be identified are present together. It has not been put into practical use in earnest due to indistinguishable problems.

On the other hand, in recent years, research for incorporating machine vision into augmented reality is being actively conducted. Mixed Reality (MR) combines reality and virtual to create a new environment in which real and virtual objects coexist, allowing users to experience various digital information more realistically by interacting with the new environment in real time. It's technology. Mixed reality (MR) includes augmented reality (AR) that adds virtual information based on reality and augmented virtuality (AV) that adds reality information to a virtual environment.

A virtual touch method and apparatus using a 3D camera disclosed in Korean Patent Publication No. 2013-0050672 is a technology that uses both machine vision and augmented reality (AR), and detects the shape of the user and converts the touch operation by the user into 3D. It is a method of realizing a virtual touch method like a real touch screen without a touch display or a special touch recognition device using a recognized 3D camera.

However, since it detects the shape of the user's hand and the pattern of movement, if the angle or movement pattern that the camera cannot recognize is not clear, or if the shape of the user's shadow is reflected on the screen, it is incorrectly recognized and an error in virtual touch. Can occur.

An object of the present invention is to provide a user interface scheme for accurately identifying an object in a content image regardless of the brightness of a place where interactive content is executed.

Another object of the present invention is to provide a user interface scheme that provides compatibility without additional modification to interactive content applications distributed on the market.

An embodiment of the present invention for achieving the above object, a digital camera for photographing a virtual interactive content image displayed on a wall; And an object recognition module that identifies a predefined object in the captured image of the virtual interactive content and determines the distance and coordinates of the object, and an event including the coordinates of the object when the object hits a wall surface to the interactive content application. It relates to a system for providing a user interface of virtual interactive content including an application driving device that executes a conversion engine including an event module to deliver.

The system of this embodiment may further include an image output device that displays the image of the virtual interactive content on the wall.

In addition, the system of this embodiment further includes a machine learning server that repeatedly analyzes a plurality of image data including the object to learn a pattern related to at least one of a shape, size, surface pattern, and color for identifying the object. Can include.

The digital camera of the present embodiment may have at least two image sensors, and in this case, the object recognition module may calculate a distance between the digital camera and the object by using a difference in angle of view of the image sensors.

The digital camera according to the present embodiment may have at least one image sensor, and in this case, the object recognition module may calculate a distance between the digital camera and a wall surface based on the size of an object in an image captured by the digital camera.

Another embodiment of the present invention includes the steps of identifying a pre-learned object from a captured image of virtual interactive content; Determining the distance and coordinates of the identified object; Generating an event including coordinates of a touch point when the object hits a wall; And transmitting the event to a virtual interactive content application.

The method of providing a user interface of the present invention may further include determining that the object has touched the wall when the calculated distance of the object matches the preset distance of the wall.

In addition, the method for providing a user interface of the present invention is a machine that repeatedly analyzes a plurality of image data including the object to learn a pattern related to at least one of a shape, a size, a surface pattern, and a color to identify the object. It may further include a running step.

Another embodiment of the present invention relates to a computer program in which the above-described method for providing a user interface is implemented as an algorithm or a computer-readable recording medium in which the program is stored.

According to an embodiment of the present invention, it is possible to enjoy sports interactive content without being affected by environmental factors of a play place such as illumination, temperature, and humidity. For example, content can be enjoyed comfortably in an indoor space with sufficiently bright lighting even on hot, cold, or high concentration of fine dust, and content can be enjoyed on an outdoor court in an area where the temperature and weather suitable for exercise are maintained.

In addition, according to an embodiment of the present invention, the recognition rate can be remarkably improved by learning in advance various characteristics of a throwing object that serves as a mouse that controls execution of content through repetitive analysis.

Further, according to an embodiment of the present invention, since the conversion engine generating the event and the virtual interactive content receiving the event are independently executed, there is no need to modify the virtual interactive content to maintain compatibility between the two programs. Therefore, the productivity of the interactive content is increased while the universality of the conversion engine is guaranteed.

1 is a conceptual diagram schematically showing a configuration of a system for providing a user interface according to a first embodiment.

2 is a block diagram of a system for providing a user interface according to the first embodiment.

3 and 4 are block diagrams showing the system configuration of a modified embodiment of the first embodiment.

5A to 5D illustrate photographing scenes of an object image for machine learning.

6 is a block diagram of a system for providing a user interface according to a second embodiment.

7 is a flow chart showing step-by-step a method of providing a user interface according to the third embodiment.

8 is a flowchart illustrating a machine learning process step by step in a method of providing a user interface according to the third embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

The terms used in the present specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, processes, operations, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the possibility of addition or presence of elements or numbers, processes, operations, components, parts, or combinations thereof is not preliminarily excluded.

Unless otherwise defined, all terms including technical or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

The term "MODULE" described herein refers to a unit that processes a specific function or operation, and may mean hardware or software, or a combination of hardware and software.

Terms and words used in the present specification and claims are not limited to their usual or dictionary meanings and should not be interpreted, and the inventor may appropriately define the concept of terms in order to describe his own invention in the best way. Based on the principle that there is, it should be interpreted as meaning and concept consistent with the technical idea of the present invention. In addition, unless there are other definitions in the technical terms and scientific terms used, they have the meanings commonly understood by those of ordinary skill in the art to which this invention belongs, and the gist of the present invention in the following description and accompanying drawings Descriptions of known functions and configurations that may be unnecessarily obscure will be omitted. The drawings introduced below are provided as examples in order to sufficiently convey the spirit of the present invention to those skilled in the art. Accordingly, the present invention is not limited to the drawings presented below and may be embodied in other forms. In addition, the same reference numbers throughout the specification indicate the same elements. It should be noted that the same components in the drawings are indicated by the same reference numerals wherever possible.

In the specification of the present invention, the term "moving object" or "object" refers to an object that can cause movement by a user using a part of his or her body or by using equipment such as a racket or a club. Volleyball ball, tennis ball, badminton ball, Ozami, darts, and the like are exemplified. However, the present invention is not limited thereto, and any object that maintains a certain shape and can be easily moved by a user may correspond to a “object”. These “objects” may also be referred to as “virtual mouse objects” or “virtual pointer objects” in that they serve as input means (eg, mouse, pointer, etc.) for executing or controlling virtual interactive content.

In the specification of the present invention, the term "interactive content" refers to content that outputs or executes various results in response to a user's real-time action, not content that is unilaterally played or executed according to a predetermined plot. .

In addition, "virtual interactive content" does not execute the content using conventional input means such as a mouse or touch pad (hereinafter referred to as'mouse, etc.'), but the actual content is executed on a separate computer device. However, the execution image of the content is directly projected on a wall, floor, or ceiling (hereinafter referred to as'wall surface') through a beam projector, or on a screen installed on a wall, or a display device installed on a wall (for example, It refers to interactive content that is output through a digital TV or a digital monitor) and virtually implements the same effect as an input means such as a mouse by touching a wall surface on which the image of the content is displayed through a moving object.

Such interactive content may be implemented as media content such as a movie, a digital book, or a digital picture frame, or as an interactive game performed by a user's touch input.

Embodiment 1 relates to a system for providing a user interface of virtual interactive content that recognizes a moving object using a stereo camera.

As shown in FIG. 1, the user plays the content by throwing the ball corresponding to the virtual mouse object toward a specific point on the wall where the content is displayed.

A digital camera 10 for photographing a user's action and content scene is disposed on a wall or ceiling opposite to the wall surface on which the content is projected, and the interactive content is executed by an application driving device 20 provided separately.

An image output device 30 that receives an image of interactive content from the application driving device 20 and outputs it to the wall surface is disposed on the wall or ceiling opposite the wall surface on which the content is projected.

2 is a block diagram showing a detailed configuration of a system for providing a user interface according to the first embodiment.

Referring to FIG. 2, the system of Example 1 includes a digital camera 10, an application driving device 20, and an image output device 30, and may further include a machine learning server 40.

The digital camera 10 photographs a content scene including a moving virtual pointer object, and transmits the photographed image data to the application driving device 20.

For easy data transmission, the digital camera 10 may be connected to the application driving device 20 through a wired communication interface such as USB, RJ-45, or a short-range or broadband wireless communication interface such as Bluetooth, IEEE 802.11, and LTE. . The communication interface or communication protocol mentioned here is only an example, and any communication interface and protocol for smoothly transmitting image data can be used.

A stereo-type measurement algorithm may be used to identify a moving object in image data and estimate a distance between the camera 10 and the moving object. In the stereotype technique, the same object is photographed using two camera modules (image sensors) separated from each other, and the distance to the object is estimated by using the angle difference caused by the discrepancy between the viewpoints between the two camera modules.

Since the system of Example 1 uses a stereo type technique, the digital camera 10 of Example 1 includes at least two 2D image sensor modules (not shown).

Next, the application driving device 20 executes the conversion engine 21 and the interactive content application 22.

The application driving device 20 may install and execute the conversion engine 21 and the interactive content application 22 together in a single device such as a desktop PC, a notebook computer, a mobile tab, a smartphone, and a server.

Alternatively, the application driving device 20 may install and execute the conversion engine 21 on a single device such as a desktop PC illustrated above, and install and execute the interactive content application 22 on a separate server 20-1. have. 3 is a block diagram showing the system configuration of such a modified embodiment.

Alternatively, the conversion engine 21 is installed and executed on the digital camera 10, and only interactive content applications are executed on the application driving device 20, and the digital camera 10 and the application driving device 20 are It can be connected through a local area network or an LTE or 5G broadband network. 4 is a block diagram showing the system configuration of this modified embodiment.

The transformation engine 21 generates an event corresponding to a click of a mouse when the moving object collides with the wall, and transmits the event to the interactive content application 22. To this end, the conversion engine 21 may include an object recognition module 21-1 and an event module 21-2.

The object recognition module 21-1 identifies a moving object by processing the image data sent from the camera 10, and estimates the distance between the camera 10 and the object using a stereotype technique. Object identification and distance estimation will be collectively defined as tracking. Tracking may be performed on all frames of image data sent from the camera 10, or intermittently performed on frames of preset intervals in consideration of the burden of load of the conversion engine 21 due to frequent tracking. It could be.

In addition, the object recognition module 21-1 may be included in the conversion engine 21 or installed in the digital camera 10 as firmware. When installed as firmware in the digital camera 10, the digital camera 10 provides tracking information including the distance to the object and the coordinates of the object instead of image data to the event module 21-2 of the conversion engine 21 do.

The event module 21-2 determines whether the moving object collides with the wall, converts the coordinates of the collision point into coordinates on the execution screen of the interactive content application, generates an event including the converted coordinates, and interactively generates an event. Send to the content application.

The principle of the event module 21-2 determining whether a moving object has collided with a wall surface may be implemented with various algorithms.

An example algorithm is as follows. That is, the distance A between the camera 10 and the wall surface is measured in advance and stored in the conversion engine 21. The event module 21-2 compares the distance (B) with the object continuously sent by the object recognition module 21-1 with the previously stored distance (A), and when the two distances (A, B) become the same, the object Is considered to have hit the wall.

Another example algorithm is as follows. That is, the event module 21-2 continuously monitors the change in the distance B with the object sent from the object recognition module 21-1. And the moment when the distance B increases and then turns to a decrease is determined as the moment of collision.

Another example algorithm is as follows. That is, the event module 21-2 continuously monitors the change in the size of the object identified in the image data sent from the object recognition module 21-1. Since the size will gradually decrease as the distance from the camera 10 increases, the moment when the size of the object decreases and then turns to increase is determined as the moment of collision.

The three algorithms mentioned above are only examples, and those skilled in the art will be able to determine whether an object collides using various principles.

The event module 21-2 has a mapping table in which the XY coordinates of the wall screen on which the content image is actually displayed and the xy coordinates on the execution screen of the content application are matched in advance.

When it is determined that the moving object collides with the wall, the event module 21-2 finds the XY coordinate of the collision point by processing the image data, and finds the xy coordinate matching the XY coordinate from the mapping table. Here, the mapping table may be a database in which XY coordinates at predetermined intervals and xy coordinates at predetermined intervals are stored in advance, or an algorithm defining a correlation between the XY coordinates and the xy coordinates by an equation.

The event module 21-2 generates an event including the converted xy coordinate and transmits it to the interactive content application.

Graphical user interface (GUI)-based operating systems such as Microsoft's Windows or Apple's MAC OS, and applications running on those operating systems receive user instructions in a so-called event driven method. .

For example, when the user positions the mouse at the coordinate point of (An, Bn) on the application screen and then clicks the left mouse, the operating system starts with mouse_move_Event(A1,B1), mouse_move_Event(A2,B2) including the coordinates of the mouse. , mouse_move_Event(A3,B3)… By continuously generating the mouse cursor (A1,B1), (A2,B2), (A3,B3)... It is moved to the path of and displayed, and by generating mouse_left_Click(An, Bn) at the point where the mouse is stopped, it notifies the operating system or the activated application that the left mouse button is clicked at the coordinates of (An, Bn).

In all embodiments of the present invention including the first embodiment, the term “event” should be understood as a concept including all events for inputting a user's instruction to the interactive content application 220. Accordingly, events transmitted from the conversion engine 21 to the interactive content application 220 may be variously defined as a left mouse click event, a right mouse click event, a mouse movement event, a mouse double click event, and a mouse wheel click event.

As a specific example, when the object recognition module 21-1 identifies a plurality of objects, when the first object is recognized by the object recognition module 21-1, the left mouse click event is performed by the event module 21-2. Is generated, a mouse right-click event is generated when the second object is recognized, and a mouse wheel click event is generated when the third object is recognized. In this case, since the player can control the virtual interactive content using three types of objects, it is possible to enjoy content with a richer plot.

The present invention makes a moving object operate like a mouse or a pointer through a method in which the conversion engine 21 generates an event and transmits the generated event to the interactive content application 22.

The event generated by the conversion engine 21 is compatible with the operating system in which the interactive content application 22 is executed. Alice, the developer of the interactive content application 22, does not need to discuss compatibility with Bob, the developer of the conversion engine 21 in advance, so the conversion engine 21 of the present invention is sold on the market. It has the advantage of being able to apply any interactive content to be applied without a separate modification for interfacing.

Next, the image output device 30 may be any type of device as long as it has a function of outputting a content image on a wall or the like.

For example, a beam projector, a display device such as a large TV or monitor mounted on a wall, and an augmented reality headset may be used as the image output device 30. The image output device 30 is connected to the application driving device 20 through a cable or wireless communication.

When an image is output using a beam projector, a problem may occur such as a shadow on the image by a user moving an object. In this case, by irradiating the same image from different angles with a plurality of beam projectors, an image without a shaded area by a user may be displayed.

Finally, the machine learning server 40 includes a machine learning engine (not shown) that learns various characteristics for identifying an object based on the image data sent from the camera 10.

For example, if the object to be recognized is a soccer ball, the machine learning server 40 uses a certain number of characteristics to identify the object based on at least one of the shape of the ball, the size of the ball, the pattern pattern on the surface of the ball, and the color of the ball. Patterns can be found.

The machine learning server 40 may receive image data through an application driving device 20 connected to the digital camera 10 or may be directly connected to the digital camera 10 to receive image data.

5A to 5D illustrate examples of photographing an object at various locations in order to pre-learn identification information of an object by machine learning.

As shown in FIGS. 5A to 5D, in the machine learning step, the user places an object such as a ball on his hand, and changes the orientation of the front, rear, left, right, up and down based on the camera 10 to view dozens to hundreds of images. Take a picture. 5A to 5D illustrate a case in which the user directly grabs the object and shoots one by one, but is not limited thereto, and the object (ball) is thrown into the shooting area of the camera 10 or the user The scene of throwing the (ball) onto the wall can be recorded as a video, and machine learning can be performed on the video of each frame constituting the video.

The machine learning server 40 finds a specific pattern to more clearly identify an object by repeatedly analyzing dozens to hundreds of different image data captured in this way.

The object recognition module 21-1 of the transformation engine 21 can easily identify an object from image data using identification pattern information, which is a result obtained by learning in advance by the machine learning module 40.

In particular, in the first embodiment, since a stereo camera 10 having at least two image sensor modules (not shown) is adopted, machine learning may be performed on a 3D image of an object. Therefore, in the step of playing interactive content after machine learning, even if a 2D image similar to an object is included in the captured image of the camera or in the displayed content image, the object recognition module 21-1 of the transformation engine is Images can be accurately identified.

Meanwhile, the machine learning server 40 may learn only one object, but may learn in advance to identify a plurality of different objects when control is required with a plurality of objects according to the type of content.

Embodiment 2 relates to a system for providing a user interface of virtual interactive content that recognizes a moving object using a mono camera.

Example 2 assumes that a mono camera such as a closed-type camera (CCTV) is already installed for security purposes, or a case of adopting a mono camera to construct a system for providing a user interface relatively inexpensively is assumed. It is not necessarily limited to these cases.

6 is a block diagram showing a detailed configuration of a system for providing a user interface according to a second embodiment.

Referring to FIG. 6, the user interface providing system according to the second embodiment includes a digital camera 100, an application driving device 200 and an image output device 300, and may further include a machine learning server 400. .

The digital camera 100 photographs a content scene including a moving virtual pointer object and transmits the photographed image data to the application driving device 200.

The connection structure or communication protocol between the digital camera 100 and the application driving device 200 is the same as that of the digital camera 10 of the first embodiment.

The digital camera 100 identifies a moving object from image data and uses a structured pattern measurement algorithm to estimate a distance between the camera 100 and the moving object.

The digital camera 100 of the structured pattern technique includes at least one light projection module and at least one image sensor module, and when the light projection module projects a structured set of light patterns onto an object, the image sensor is reflected by the projection. Optical 3D scanning is performed by capturing an image, and a distance between the camera 100 and an object is measured using the 3D scanning result.

The application driving device 200 executes the conversion engine 210 and the interactive content application 220. It is the same as described in the first embodiment that the conversion engine 210 and the interactive content application 220 may be executed in one device 200 or separately executed in a separate device.

The transformation engine 210 generates an event corresponding to a click of a mouse when the moving object collides with the wall, and transmits the event to the interactive content application 220. To this end, the conversion engine 210 may include an object recognition module 211 and an event module 212.

The object recognition module 211 processes image data sent from the camera 100 to identify a moving object, and estimates the distance between the camera 100 and the object using a structured pattern technique.

The event module 212 determines whether the moving object collides with the wall, converts the coordinates of the collision point into coordinates on the execution screen of the interactive content application, generates an event including the converted coordinates, and converts the event into the interactive content application. Transfer to.

The principle of the event module 212 transforming the coordinates is the same as described in the first embodiment.

The image output device 300 and the machine learning server 400 are also the same as the image output device 30 and the machine learning server 40 of the first embodiment.

Embodiment 3 relates to a method of providing a user interface for virtual interactive content.

For convenience of explanation, a virtual interactive content image is displayed on the wall by an image output device such as a beam projector, and the user throws a virtual mouse object to the wall to play the content.

However, such a premise is only intended to help understanding, and it is a matter of course that the method of providing a user interface according to the third embodiment can be applied to various modified embodiments suggested in the first and second embodiments.

The digital camera installed on the ceiling captures an image displayed on the wall and a scene where the user throws an object on the wall, and transmits the captured image data to the application driving device in real time (S101).

The conversion engine running in the application driving device identifies the virtual mouse object learned in advance from the image data sent from the camera (S102), and tracks the movement of the object (S103). Here, “tracking” refers to a process of determining the distance between the identified object and the camera and the coordinates on the wall screen where the object is located.

When it is determined that the object hits the wall during the tracking process (S104), the conversion engine converts the XY coordinates of the touch point into the xy coordinates on the execution screen of the interactive content application (S105).

Then, a mouse event including the converted coordinates is generated, and the mouse event is transmitted to the interactive content application (S106).

For better understanding, the situation of FIG. 5 in which a user enters the shooting range of a digital camera and holds a virtual mouse object such as a ball in one hand and performs test shots tens to hundreds of times will be described in detail with reference to the situation of FIG.

The machine learning server receives image data from a digital camera or an application driving device connected to the digital camera (S201), and processes the image data to derive at least one characteristic of the shape, size, surface pattern, and color of the object ( S202).

If a certain pattern for identifying an object is defined based on the derived characteristics (S203), the machine learning process is terminated, and the defined identification pattern is provided to the transformation engine (S204) so that it can be used as reference data for object identification later. To be. And if it is still insufficient to define a certain pattern, steps S201 to S203 are repeatedly performed.

A user can take tens to hundreds of images while placing an object such as a ball on his hand and changing the orientation of the front, rear, left, right, up and down directions based on the camera. The machine learning server repeatedly analyzes dozens to hundreds of different image data captured in this way, thereby defining a specific pattern to more clearly identify an object.

The termination of the machine learning process may be automatically executed when a preset criterion is satisfied, or may be executed arbitrarily at the discretion of an administrator.

The pattern for object identification defined through the above steps is provided to the conversion engine, so that the object can be accurately identified even if there is any kind of background in the still image of the moving object.

The entire or partial functions of the method for providing the user interface of the virtual interactive content of the third and fourth embodiments described above are provided in a recording medium that can be read through a computer by tangibly implementing a program of instructions for implementing this. It will be readily understood by those skilled in the art that it may be. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and constructed for the present invention, or may be known and usable to those skilled in computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. Magneto-optical media and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, USB memory, and the like. The computer-readable recording medium may be a transmission medium such as an optical or metal wire or a waveguide including a carrier wave for transmitting a signal specifying a program command or a data structure. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like, in addition to machine language codes such as those produced by a compiler. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention and vice versa.

In addition, the present invention is not limited to the above-described embodiments, and of course, various modifications can be implemented without departing from the gist of the present invention as claimed in the claims, as well as various application ranges.

Claims

A digital camera that photographs a virtual interactive content image displayed on a wall; And

An object recognition module that identifies a predefined object in the captured image of the virtual interactive content and identifies the distance and coordinates of the object, and delivers an event including the coordinates of the object to the interactive content application when the object hits a wall Application driving device that executes a conversion engine including an event module

A system for providing a user interface of virtual interactive content including a.
The method of claim 1,

A system for providing a user interface of virtual interactive content, further comprising an image output device that displays the image of the virtual interactive content on a wall.
The method of claim 2,

The video output device,

A system for providing a user interface for virtual interactive contents, characterized in that it is any one of a beam projector, a display device mounted on a wall, and an augmented reality headset.
The method of claim 1,

The virtual interactive content further comprises a machine learning server that repeatedly analyzes a plurality of image data including the object to learn a pattern related to at least one of a shape, size, surface pattern, and color for identifying the object. User interface providing system.
The method of claim 1,

The digital camera has at least two image sensors,

The object recognition module calculates a distance between the digital camera and the object using a difference in angle of view of the image sensors.
The method of claim 1,

The digital camera has at least one image sensor,

The object recognition module calculates a distance between the digital camera and the object based on the size of the object in the image captured by the digital camera.
Identifying an object learned in advance from the captured image of the virtual interactive content;

Determining the distance and coordinates of the identified object;

Generating an event including coordinates of a touch point when the object hits a wall; And

Delivering the event to a virtual interactive content application

Method for providing a user interface of virtual interactive content comprising a.
The method of claim 7,

The photographed image is photographed by a digital camera having at least two image sensors,

The distance is calculated based on a difference in angle of view of the image sensors.
The method of claim 7,

The photographed image is photographed with a digital camera having one image sensor,

The distance is calculated based on the size of an object in the content image.
The method of claim 8,

And determining that the object has touched the wall when the calculated distance of the object matches the preset distance of the wall.
The method of claim 7,

The virtual interactive content further comprising a machine learning step of learning a pattern related to at least one of a shape, a size, a pattern pattern on a surface, and a color for identifying the object by repeatedly analyzing a plurality of image data including the object. How to provide a user interface.
A computer-readable recording medium storing a computer program that implements the method of claims 7 to 11 as an algorithm.