KR20190094875A - Digital signage system for providing mixed reality content comprising three-dimension object and marker and method thereof - Google Patents

Abstract

Disclosed is a digital signage system for providing mixed reality (MR) contents comprising a three-dimensional object and markers, and a method thereof. According to the present invention, MR contents are produced by adding one or more markers based on a three-dimensional object generated through an exclusive tool or a three-dimensional object through object recognition of an image photographed through a camera, and the produced MR contents are provided through the digital signage device associated with sight, hearing, olfaction, etc. A variety of additional information associated with the three-dimensional object is provided and virtual experience and simulation functions of products are provided. Accordingly, satisfaction of a user can be increased and utilization of the MR contents according to requests of the user can be widened.

Description

Signage system and method for providing mixed reality content comprising three-dimension object and marker and method

The present invention relates to a signage system and method for providing MR content including 3D objects and markers, and more particularly, to a 3D object generated through a dedicated tool or a 3D object through object recognition in an image captured by a camera. Signage system for creating MR content by adding one or more markers as a basis, and providing MR content including markers and 3D objects that provide the produced MR content through signage devices related to visual, auditory, olfactory, auditory, etc. And to a method thereof.

Digital signage is a communication tool that can induce marketing, advertising, training effects, and customer experiences of companies. It is a digital image display device that provides specific information in public places such as airports, hotels, and hospitals.

The current advertising method using digital signage is difficult to remember for a long time even if the user is just overlooking or interested in the content encountered through digital signage, and it is difficult to check additional information on the content. In particular, this problem is more prominent when the digital signage is installed in a place where there are many users and are frequently moved, such as a street or an underground walkway.

Korean Registered Patent No. 10-1619886 [Name: Digital Signage System and Mobile Interworking Method]

An object of the present invention is to create an MR content by adding one or more markers based on the 3D object generated through the dedicated tool or the object recognition in the image captured by the camera, and the MR content A signage system and method for providing MR content including 3D objects and markers provided through a signage device related to visual, auditory, olfactory, auditory, and the like are provided.

A signage method for providing MR content including a 3D object and a marker according to an embodiment of the present invention is a signage method for providing MR content including a 3D object and a marker. Outputting MR content provided from a server based on the time information; Recognizing, by the digital signage, a voice of a user; And when the marker included in the output MR content is selected by the digital signage based on the recognized voice of the user, performing a function corresponding to the selected marker.

The present invention creates MR content by adding one or more markers based on the 3D object generated through a dedicated tool or 3D object through object recognition in an image captured by a camera, and visualizes the produced MR content. By providing through signage devices related to hearing, smell, hearing, etc., it provides various additional information related to 3D objects, improves user satisfaction by providing virtual experience and simulation of products, and MR contents according to user needs. It is effective to widen the range of use.

1 is a block diagram showing the configuration of a signage system for providing MR content including a 3D object and a marker according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a digital signage according to an embodiment of the present invention.
3 is a diagram illustrating a concept of element technology included in a digital signage according to an exemplary embodiment of the present invention.
4 is a flowchart illustrating a signage method for providing MR content including 3D objects and markers according to an embodiment of the present invention.
5 to 8 are diagrams illustrating a screen of a terminal according to an embodiment of the present invention.
9 is a diagram illustrating a screen of a digital signage according to an embodiment of the present invention.

It should be noted that the technical terms used in the present invention are merely used to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be interpreted as meanings generally understood by those skilled in the art unless the present invention has a special meaning defined in the present invention, and is excessively comprehensive. It should not be interpreted in the sense of or in the sense of being excessively reduced. In addition, when a technical term used in the present invention is an incorrect technical term that does not accurately express the spirit of the present invention, it should be replaced with a technical term that can be understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced sense.

Also, the singular forms used in the present invention include plural forms unless the context clearly indicates otherwise. Terms such as “consisting of” or “comprising” in the present invention should not be construed as necessarily including all of the various components or steps described in the present invention, and some of the components or some steps may not be included. It should be construed that it may further include, or further include, additional components or steps.

In addition, terms including ordinal numbers such as first and second used in the present invention may be used to describe components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted.

In addition, in describing the present invention, when it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

1 is a block diagram showing the configuration of a signage system 10 for providing MR content including 3D objects and markers according to an embodiment of the present invention.

As shown in FIG. 1, the signage system 10 includes a terminal 100, a server 200, a digital signage 300, and another terminal 400. Not all components of the signage system 10 shown in FIG. 1 are essential components, and the signage system 10 may be implemented by more components than those shown in FIG. Signage system 10 may also be implemented by components.

The terminal 100 and the other terminal 400 are a smart phone, a portable terminal, a mobile terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ) Terminals, Telematics Terminals, Navigation Terminals, Personal Computers, Notebook Computers, Slate PCs, Tablet PCs, Ultrabooks, and Wearable Devices For example, a watch type terminal (Smartwatch), a glass type terminal (Smart Glass), a head mounted display (HMD), etc.), a Wibro terminal, an Internet Protocol Television (IPTV) terminal, a smart TV, a digital broadcasting terminal The present invention can be applied to various terminals such as an audio video navigation (AVN) terminal, an audio / video (A / V) system, a flexible terminal, and the like.

In addition, the terminal 100 is an image including an object photographed using a dedicated tool (or dedicated program) pre-installed in the terminal 100 or through a camera unit (not shown) provided in the terminal 100. The information is used to create (or create / model) a 3D object (or 3D object data related to the 3D object).

That is, the terminal 100 generates (or fabricates / models) a 3D object related to a specific object by using a dedicated tool (or dedicated program). Here, the dedicated tool (or dedicated program) may be any one of known tools (or programs / applications) capable of generating 3D objects related to any specific object by user's manipulation (or user's control). .

In the embodiment of the present invention, the description has been mainly focused on 3D objects, but is not limited thereto and may include 2D objects.

The 3D object or 2D object may include not only an image (or still image) but also a moving picture, a 360VR image, a panoramic image, etc. to which the 3D object / 2D object is applied.

In addition, the terminal 100 recognizes an object in an image (or image information) including an arbitrary object photographed (or obtained) through a camera unit included in the terminal 100, and recognizes an object in the terminal 100. A 3D object is generated (or manufactured / modeled) using the recognized object through a pre-installed 3D object generation program (not shown). In this case, the image may be provided from any terminal (not shown) or server 200 communicating with the terminal 100. In addition, the 3D object generation program of the known program (or tools / applications) that can create (or convert / produce) the object contained in any image as a 3D object by the user's operation (or user's control) It can be either.

Here, the generated (or manufactured / modeled) 3D object (or 3D object data) is generated to include visual information, auditory information, tactile information, olfactory information, and the like related to the 3D object, and thus, for the corresponding 3D object. When providing information, it can be configured to simultaneously provide information on visual, auditory, tactile, olfactory sense. In this case, the visual information is generated to include shape information, color information, material information, pattern information, design information, etc. of the 3D object, so that accurate and various information can be provided to the user when providing information related to the 3D object. . The auditory information may include sound information related to the corresponding 3D object. In addition, the tactile information may include information about the kinesthesia associated with the 3D object (for example, information about mass, firmness, and reaction force associated with the 3D object), and texture related to the 3D object. ), Including information about surface patterns, small shapes, softness and roughness of the surface, and a sense of coldness associated with the 3D object. In addition, the olfactory information includes fragrance (or fragrance) information related to the 3D object.

In addition, the terminal 100 is configured to provide the generated (or produced / modeled) 3D object to the server 200, integrated management through a database (not shown) provided in the server 200 You may.

In addition, the terminal 100 may utilize the generated 3D object in virtual reality (or augmented reality / mixed reality) as well as a distribution industry such as a virtual store, inventory management, and counterfeit control.

In addition, in order to insert a marker (or 3D marker / object marker) associated with the generated (or manufactured) 3D object, a preset marker insertion button displayed on one side of the screen of the terminal 100 in which the image is displayed ( Alternatively, when a menu / item) (or a marker insertion button provided on one side of the terminal 100) is selected, the terminal 100 arranges a marker (or an object marker) adjacent to the generated 3D object. Accordingly, the terminal 100 displays the marker in a state spaced apart from the object by a predetermined distance on one side of the image including the corresponding object.

In addition, after a preset marker insertion button displayed on one side of the screen of the terminal 100 is selected (or dragged) by using a drag and drop function, the terminal ( When dropped over a 3D object being displayed on the screen of 100) (or when dropped to an area / position within a preset radius from the 3D object), the terminal 100 is separated from the 3D object by a predetermined distance. The marker may be arranged at a position (or a specific position).

In this case, the terminal 100 is arranged to be spaced apart by a predetermined interval with respect to the 3D object, and the marker is set to maintain a constant interval on the 3D object even if the 3D object is rotated or reduced / enlarged in three dimensions (Or layout / design).

That is, the terminal 100 configures a layer in which the 3D object operates and a layer in which the marker operates (or displays) differently, so that the marker may also be rotated, reduced, enlarged, or moved. May be set to maintain a spaced state from the 3D object by a predetermined interval.

In addition, the terminal 100 may arrange the marker to overlap on the 3D object.

That is, the terminal 100 may further form an upper layer on the screen on which the 3D object is displayed, and place the marker on the additionally formed upper layer.

In an embodiment of the present invention, in order to configure the marker and the 3D object to be controlled independently, it is described that the 3D object and the marker are formed on different layers, but the present invention is not limited thereto. And various techniques in which the 3D object may independently operate (or control) may be applied.

An image including the marker and / or the object described in the embodiment of the present invention may be a 2D or 3D image, a still image, a video, a 360VR image, a panoramic image, an aerial photograph (or aerial image), a satellite image (or a satellite image). ), Skyview, roadview, and so on. Here, the aerial photograph (or aerial image) includes a still image or a moving image photographed at a predetermined altitude from the ground through an aircraft, a drone, or the like. In addition, the satellite image (or satellite image) or the sky view includes a still image or a video photographed at a predetermined altitude from the ground through a satellite or the like. In addition, the road view is equipped with a camera (including a camera capable of shooting 360 degrees, etc.) in a vehicle, motorcycle (bibicycle), and other means of transportation, and photographed the surroundings of the vehicle, etc. as the vehicle moves It includes still images or moving pictures.

In addition, the aerial photograph (or aerial image), satellite image (or satellite image), sky view, road view, etc. store any location information through the GPS at the time of shooting with the captured image, any object included in the captured image (E.g., including buildings, one or more businesses located within the building, etc.) can be accurately and easily identified, and markers associated with the identified objects may be added.

As such, the 3D object and the marker are formed on different layers, and the marker is configured to be controlled independently of the 3D object, so that the marker is fixed from the 3D object even when the 3D object is moved, enlarged or reduced. It may be configured to be spaced apart or positioned on the 3D object.

In addition, when a video (or 360 VR image / panorama image) is constructed using the 3D object, the marker is located at or adjacent to the 3D object (or 2D object) while the 3D object is displayed in the video. The display is overlapped with an object, and the marker may be disappeared when the 3D object disappears in the video. In addition, when a new 3D object (or a new 2D object) appears in the video, a new marker may be displayed corresponding to the new 3D object.

In addition, when the marker being displayed adjacent to or overlapping with the 3D object is selected, an additional screen may be displayed according to the setting information set for the marker.

For example, when the marker being displayed adjacent to or overlapping with the 3D object is selected, a screen for viewing detailed information related to the marker (or related to the 3D object corresponding to the marker), and for payment function A screen, a screen for sharing function, an enlarged screen of the corresponding 3D object (or 2D object) related to the marker, or another image / screen / video related to the 3D object (or 2D object), a screen for performing a chatbot function, and the like. It may be configured to be displayed in a popup form on the screen or to be displayed on the entire screen.

Here, a chatbot (chatterbot, conversation robot) is a program or robot made for a conversation between a person and a machine, the chatbot function is a user and the server 200 or the digital signage 300 looking at the screen It provides an interactive user interface system for providing a robot to send and receive information in a dialog, including artificial intelligence (AI) function, the user (or customer) questions related to the object corresponding to the selected marker Include the ability to automatically respond to AI application (including price bargains, inventory inquiries, delivery inquiries, performance inquiries, usage instructions, etc.).

In addition, the embodiment of the present invention has been described that the marker is configured in a fixed form, but is not limited thereto, and the marker may be configured in a 2D or 3D form, and may be a reduced form of a corresponding 3D object or a designer. May be arbitrarily designed (or designed) by, or animation effects (including, for example, rotation, flicker, bounce, etc.) may be applied.

For example, in a state in which a refrigerator store is configured in a 3D form, a 3D marker in which each refrigerator is reduced for a plurality of refrigerators configured in the refrigerator store may be displayed adjacent to or overlapping with each of the plurality of refrigerators. In addition, an animation function (for example, a function of rotating a refrigerator image reduced from right to left) may be applied to the 3D marker.

As such, as the animation effect is applied to the marker, the marker displayed together with the 3D object (or 2D object) is more prominent by the user, and thus the case of selecting the marker increases. Optionally, additional functions (or additional information) may be provided.

In addition, the markers may be displayed by solid-line processing the edges of the markers to distinguish them from the temporary markers described below, and the temporary markers may be displayed by dotted lines. In this case, the marker is included in Mixed Reality Contents (or MR image), and includes information on at least one other MR content linked to (or matching with) an object included in the MR content. In addition, the temporary marker may be an indication (or information) indicating that MR content related to an object included in an image acquired through the camera unit exists in the terminal 100 and / or the server 200. .

In addition, when the marker displayed on the terminal 100 is selected (or touched / double-clicked), the terminal 100 may provide attribute information on the selected marker based on a user input (or user touch / user selected). Set it. Here, the attribute information (or the attribute information of the marker) for the marker may include unique code information of an object (or 3D object) related to the marker, unique code information of the marker related to the object, and at least one MR content related to the object. Information about (or address information / link address where at least one MR content related to the object is stored), visual information, auditory information, tactile information, olfactory information, etc., related to the 3D object corresponding to the marker. In this case, the at least one MR content related to the object may be MR content corresponding to at least one other 3D object having the same code or similar code with respect to the object.

In addition, when setting information on at least one MR content related to the object, the terminal 100 is linked to the server 200, the user (or specific / arbitrary) of the terminal 100 through machine learning User information (e.g., preferences, purchase history information, information about the products that users searched for the most for that object, and the most sold items for that object) Information, promotion information, and big data related to the object and / or the user), among at least one MR related to the object among a plurality of MR contents stored in the terminal 100 and / or the server 200 in advance. The content can be selected (or set).

As such, the terminal 100 accumulates the user information so as to provide customized MR content optimized for the corresponding user (or a specific / arbitrary user), and machine learning about the accumulated user information. Through the information related to the user (or a specific / arbitrary user) may be set through the attribute information for the selected marker of the adaptive conversion form.

In addition, when a preset preview menu (or button / item) displayed on one side of the terminal 100 is selected, the terminal 100 displays a result image of MR content creation as a preview screen. In this case, the terminal 100 may display the resultant image of the MR content authoring on a new screen or the resultant image of the MR content authoring in a pop-up form.

That is, when the preview menu is selected, the terminal 100 displays a result image of the MR content creation (or an image including the generated object and a marker disposed adjacent to the object) in a preview form. . In this case, the resultant image of the MR content authoring may include an immediate purchase menu for providing a purchase function related to the object, a sharing menu for providing a sharing function for the MR content, and the like.

In addition, when a preset storage menu (or completed menu / button / item) displayed on the other side of the terminal 100 is selected, the terminal 100 is a 3D object, a marker disposed adjacent to the 3D object. Create MR content including the like. In this case, the MR content is a corresponding 3D object, property information (or product information) of the 3D object, code associated with the 3D object, unique identification information of the code, type of code, marker corresponding to the 3D object, marker location information, Property information of a marker (including, for example, information on at least one MR content related to the corresponding 3D object, visual information related to the 3D object corresponding to the marker, auditory information, tactile information, olfactory information, etc.) Immediately include purchase information, shared information and more.

That is, when the preset storage menu is selected, the terminal 100 checks whether the attribute information setting of the marker according to the user selection is completed for the marker.

In addition, when the check result, the property information setting for the marker is not completed, the terminal 100 indicates that the property information setting of the marker is not completed, and the property information setting of the marker The guide information is displayed on one side.

In addition, as a result of the checking, when the attribute information setting for the marker is completed, the terminal 100 generates MR content including the 3D object, a marker related to the 3D object, and the like.

As such, the terminal 100 may generate MR content by mixing an image including an actual object and an image including a marker for providing information related to the object.

In this case, when the preset storage button is selected while a video is generated using the generated 3D object, the terminal 100 identifies unique identification information of a code related to the 3D object and a marker corresponding to the 3D object. (Or match / interlate / update) each of the markers so as to be adjacent to the 3D object included in the video, based on (or unique identification information of the code related to the area / location where the marker is placed). MR content in the form can be generated.

As such, when the image including the object is not only a still image (or an image) but also a video, the terminal 100 may generate the MR content by inserting a marker related to the object included in the image.

In addition, as described above, the terminal 100 utilizes MR content including the 3D object and a marker related to the 3D object, an associated product recommendation function related to a product corresponding to a 3D object included in the MR content, and a product recommendation. Functions, promotions, interactions (or interactions), simulations, and the like can be provided to enhance the user's convenience of searching for and checking products and to increase user satisfaction.

In addition, when the marker is disposed on the screen of the terminal 100 or when setting of attribute information on the marker disposed on the screen of the terminal 100 is completed, the terminal 100 is connected with the marker. The object (or code related to the object) included in the related image / video is stored in association with the marker.

In this case, the terminal 100 extends the outline of the disposed marker to the outline of the object related to the marker included in the image / video. That is, the terminal 100 matches and stores the coordinate values of the outline of the arranged marker with the coordinate values of the outline of the object associated with the marker.

For example, when the marker associated with the object corresponding to the wireless vacuum cleaner is disposed, and the setting of the attribute information for the disposed marker is completed, the terminal 100 displays the coordinate value of the outline of the arranged marker. Match (or extend) the coordinate values of the outline of the vacuum cleaner.

As described above, as the outline of the marker is extended to the outline of the object related to the marker, an effect of selecting the marker may be obtained even when the corresponding object is selected later.

In addition, the terminal 100 transmits the generated MR content, identification information of the terminal 100, etc. to the server 200 in order to share the generated MR content. In this case, the identification information of the terminal 100 includes a mobile directory number (MDN), a mobile IP, a mobile MAC, a subscriber identity module (SIM) card unique information, a serial number, and the like.

The server 200 communicates with the terminal 100, the digital signage 300, the other terminal 400, and the like.

In addition, the server 200 receives MR content, identification information of the terminal 100, and the like, which are transmitted from the plurality of terminals 100, including the terminal 100. Here, the MR content may be content generated (or produced) in each terminal 100.

In addition, the server 200 stores (or manages) the plurality of received MR contents.

In addition, the server 200 is a specific digital based on a plurality of MR content stored in the server 200 (or including a separate advertising or promotional content produced in a format (or form) different from the MR content) Scheduling of the content to be output from the signage 300 is performed.

That is, the server 200 uses the plurality of MR contents stored in the server 200 (or includes a separate advertisement or promotion content produced in a format (or form) different from MR contents). For each of the digital signages 300 managed by 200, the digital signage 300 generates playback scheduling information including information on content to be output for each date and / or time zone.

In addition, the server 200 is a specific digital based on a plurality of MR content stored in the server 200 (or including a separate advertising or promotional content produced in a format (or form) different from the MR content) Scheduling of the content to be output from the signage 300 is performed.

That is, the server 200 generates playback scheduling information including information on content to be output by date and / or time zone according to time cells for a plurality of MR contents.

In addition, the server 200 corresponds to the generated playback schedule information, a plurality of MR content (or a separate advertisement or promotional content produced in a different format from the MR content, etc.) according to the corresponding playback schedule information, and the like. The digital signal is transmitted to the digital signage 300 related to the reproduction schedule information.

In this case, the server 200 transmits the generated reproduction schedule information to the digital signage 300 in advance, and then streams a plurality of MR contents to be output from the digital signage 300 according to the reproduction schedule information. By transmitting to the corresponding digital signage 300 by using a method, an event method, and the like, even if the digital signage 300 does not have a large storage capacity, the plurality of MR contents provided from the server 200 are provided. Can be output sequentially.

As described above, the server 200 includes a plurality of contents according to time cells for each of the digital signages 300 (for example, MR content and AR) for the plurality of digital signages 300 managed by the server 200. Content, VR content, video, etc.) to generate (or output) playback schedule information, and generate the playback schedule information for each digital signage.

In addition, the server 200 may sequentially provide one or more contents to each digital signage 300 according to the playback schedule information for each digital signage.

2 is a block diagram showing the configuration of a digital signage 300 according to an embodiment of the present invention.

As shown in FIG. 2, the digital signage (or digital signage device) 300 includes a communication unit 310, a storage unit 320, a camera unit 330, a sensor unit 340, a display unit 350, It is composed of a voice output unit 360, the haptic unit 370, the scent generating unit 380 and the control unit 390. Not all components of the digital signage 300 shown in FIG. 2 are essential components, and the digital signage 300 may be implemented by more components than those shown in FIG. The digital signage 300 may also be implemented by components.

3 is a diagram showing the concept of the element technology included in the digital signage 300 according to an embodiment of the present invention.

The digital signage 300 includes a multi-vision, kiosk, touch display, elevator display, outdoor digital signage, media wall, and the like.

In addition, the digital signage 300 is being developed by integrating IT technologies such as display devices, contents, networks, and software solutions, and contents-related technologies, and advanced technologies such as AR, VR, MR, AI, and Multi Modality. And digital signage are combined to expand the range of applications to various forms of smart signage.

In addition, the digital signage 300 is configured in the form of an interactive digital signage.

That is, the digital signage 300 expands network interoperability based on HTML5, applies a high-precision drilling and compatible platform (including big data analysis through a cloud system, etc.), and applies a multi-modality system (eg For example, it provides realistic information using motion control, voice recognition, and use of olfactory information, and maximizes the experience of products by applying VR / AR / MR technology to digital signage, and implements markers and 3D event system. Can be used to connect commerce functions directly.

The communication unit 310 communicates with any component inside or any at least one terminal outside via a wired / wireless communication network. In this case, the external arbitrary terminal may include the terminal 100, the server 200, the other terminal 400, and the like. Here, the wireless Internet technologies include a wireless LAN (WLAN), a digital living network alliance (DLNA), a wireless broadband (Wibro), a WiMAX (World Interoperability for Microwave Access (Wimax), and a HSDPA (High Speed Downlink Packet Access). ), HSUPA (High Speed Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS), etc. In this case, the communication unit 310 transmits and receives data according to at least one wireless Internet technology in a range including the Internet technologies not listed above. In addition, near field communication technologies include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). Ultrasound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and the like may be included. In addition, the wired communication technology may include power line communication (PLC), USB communication, Ethernet, serial communication, serial communication, optical / coaxial cable, and the like.

In addition, the communication unit 310 may mutually transmit information with any terminal via a universal serial bus (USB).

In addition, the communication unit 310 is a technical standard or a communication method for mobile communication (for example, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (EV), EV) Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (DO), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced) and the like transmits and receives a radio signal with a base station, the terminal 100, the server 200, the other terminal 400 and the like on a mobile communication network.

In addition, the communication unit 310 is controlled by the control unit 390, the reproduction schedule information transmitted from the server 200, a plurality of MR content (or produced in a format different from the MR content according to the reproduction schedule information) And separate content such as advertisements or promotions). In this case, the MR content is a corresponding 3D object, property information (or product information) of the 3D object, code associated with the 3D object, unique identification information of the code, type of code, marker corresponding to the 3D object, marker location information, Property information of a marker (including, for example, information on at least one MR content related to the corresponding 3D object, visual information related to the 3D object corresponding to the marker, auditory information, tactile information, olfactory information, etc.) Immediately include purchase information, shared information and more.

The storage unit 320 stores various user interfaces (UIs), graphical user interfaces (GUIs), and the like.

In addition, the storage unit 320 stores data and programs necessary for the digital signage 300 to operate.

That is, the storage unit 320 may store a plurality of application programs or applications driven in the digital signage 300, data for operating the digital signage 300, and instructions. have. At least some of these applications may be downloaded from an external server via wireless communication. In addition, at least some of these application programs may exist on the digital signage 300 from the time of shipment for the basic function of the digital signage 300. Meanwhile, an application program is stored in the storage 320 and installed in the digital signage 300 to be driven by the controller 390 to perform an operation (or function) of the MR content providing apparatus 100. Can be.

In addition, the storage unit 320 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD). Memory, etc.), magnetic memory, magnetic disks, optical disks, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), It may include at least one storage medium of the PROM (Programmable Read-Only Memory). In addition, the digital signage 300 may operate a web storage that performs a storage function of the storage 320 on the Internet, or may operate in connection with the web storage.

In addition, the storage 320 is a plurality of MR content (or a separate advertisement or promotion produced in a different format than MR content according to the playback schedule information, the playback schedule information received through the communication unit 310) Content, etc.).

The camera unit 330 processes an image frame such as a still image or a video obtained by an image sensor (camera module or camera) in a video call mode, a shooting mode, a video conference mode, and the like. That is, corresponding image data obtained by the image sensor is encoded / decoded according to each standard according to a codec. The processed image frame may be displayed on the display unit 350 under the control of the controller 390. For example, the camera photographs an object (or a subject) (eg, a product, a user, etc.) and outputs a video signal corresponding to the photographed image (subject image).

In addition, the image frame processed by the camera unit 330 may be stored in the storage unit 320 or transmitted to the server 200 or the other terminal 400 through the communication unit 310.

In addition, the camera unit 330 may provide the controller 390 with a panorama image (or panorama image information) acquired (or photographed) through a 360 degree camera (not shown). In this case, the 360-degree camera can take a panorama image of a two-dimensional or three-dimensional form.

In addition, the camera unit 330 is installed on one surface of the digital signage 300.

In addition, the camera unit 330 acquires (or captures) image information about a user (or a user's movement) located in front of the digital signage 300 in which the camera unit 330 is configured.

The sensor unit 340 is configured on one surface (or one side) of the digital signage 300.

In addition, the sensor unit 340 may be a user's motion (or motion) recognition function, a voice recognition function, a lip movement recognition function, and a pupil's movement recognition function located in front of the digital signage 300 in which the sensor unit 340 is configured. Function (or eye tracking function).

In addition, the sensor unit 340 may include a motion recognition module for recognizing a user's motion, a lip movement, or a pupil's movement, a voice recognition module for voice recognition, and the like.

In addition, as shown in FIG. 3, the sensor unit 340 is provided in the form of a wearable device, and is worn on the user's body (including a hand, finger, arm, leg, etc.) of the user. You can also recognize the information according to the movement.

The display unit 350 may display various contents such as various menu screens using a user interface and / or a graphic user interface stored in the storage unit 320 under the control of the controller 390. Here, the content displayed on the display unit 350 includes a menu screen including various text or image data (including various information data) and data such as icons, list menus, combo boxes, and the like. In addition, the display unit 350 may be a touch screen.

In addition, the display unit 350 may include a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. The display device may include at least one of a flexible display, a 3D display, an e-ink display, and a light emitting diode (LED).

In addition, as illustrated in FIG. 3, in the display unit 350, a plurality of display devices may configure one large screen.

In addition, the display unit 350 controls MR content provided from the server 200 based on the play schedule information and the current time under the control of the control unit 390 (or for advertising or promotion separate from MR content). Content, etc.).

In this case, the display unit 350 outputs MR content transmitted from the server 200 in real time or stores the MR content transmitted from the server 200 in consideration of the storage capacity of the storage unit 320. After temporary storage in the unit 320, the temporarily stored MR content may be output.

The voice output unit 360 outputs voice information included in a signal processed by a predetermined signal by the controller 390. Here, the voice output unit 360 may include a receiver, a speaker, a buzzer, and the like.

In addition, the voice output unit 360 outputs the guide voice generated by the controller 390.

In addition, the audio output unit 360 may include auditory information (eg, voice information and noise) included in MR content provided from the server 200 based on the playback schedule information and the current time by the controller 390. Information, etc.).

That is, the audio output unit 360 displays the MR content corresponding to the current time on the display unit 350 according to the playback schedule information by the controller 390 in advance in relation to the corresponding MR content being displayed. If the auditory information is set, the auditory information is output.

The haptic unit (or haptic interface / tactile interface) 370 is configured to be adjacent to the display unit 350.

In addition, the haptic interface (or the haptic unit) refers to a whole system that transmits a tactile sense to a user in a broad sense, and is a hardware device that generates a tactile sensation by transmitting a physical force to a user by directly contacting the user. A haptic device, a virtual environment consisting of a remote contact (Teletaction) environment or computer graphics and physical characteristics, and a function of transferring the movement of the haptic device to the virtual environment and transmitting the contact force with the virtual environment to the haptic device. It may be configured as a controller (not shown) (or the controller 390).

The ultimate purpose of the haptic interface is to allow the user to feel the same physical characteristics of the virtual environment (eg, AR environment, VR environment, MR environment, etc.) or real environment being modeled through the haptic device.

In addition, the haptic interface includes a virtual delegate representing a change in position of the haptic device in the virtual environment so that the haptic device and the virtual environment exist in one environment. If the force generated by the virtual delegate hitting the virtual object is Fe and the force felt by the haptic device is Fm, then the ideal haptic interface system is to control the two values to be equal.

In order to implement an ideal haptic interface, research on accurate modeling methods and hardware and control methods for implementing the model should be supported.

In addition, the haptic interface may be used in the same sense as the haptic device because the haptic interface is limited to only a hardware part in a narrow sense.

In addition, the haptic interface can be roughly divided into two fields in terms of hardware development, that is, in terms of developing the haptic device.

In other words, a person feels the sense of touch through two main paths, and the haptic interface is also developed in two main areas.

One of the fields of the haptic interface is related to muscle sensation, and when we feel the mass of the object, the degree of firmness and softness, the appearance of the object, and the like, the haptic interface feels the touch through the movement of muscles and joints. Since this process is physically related to force, the haptic device has a mechanical structure such as a robot shape that is advantageous for generating a force, and the haptic device made for this purpose is called a force feedback device. do.

In addition, another field of the haptic interface is related to texture, and is a phenomenon in which the skin is directly contacted with the surface when the pattern or small shapes of the surface being contacted, the softness and roughness of the surface, and the sense of warmth and the like are felt. This process consists of many complex physical elements, and the shapes of haptic devices that attempt to reproduce the texture are also very diverse. A device made for the purpose of conveying such a texture is called a texture display device.

As described above, the haptic device (or the haptic unit 370) is divided into two fields, a force feedback device and a texture presenting device, according to the purpose, and the embodiment of the present invention includes both the haptic devices of the two fields. do.

In particular, in order to provide the user with information about the texture using the haptic device, the vibrotactile display using vibration among the stimulation methods is easily applied to a place where only a simple signal is required because the stimulus is quickly transmitted. have.

In addition, since the pin-array method has a plurality of pins formed on the skin and directly stimulates them in the vertical direction, the pattern of the surface can be intuitively transmitted.

In addition, the haptic unit 370 displays the MR content corresponding to the current time on the display unit 350 under the control of the playback schedule information under the control of the control unit 390, and in relation to the corresponding MR content being displayed. When there is a preset tactile information, the tactile information is operated according to the corresponding tactile information (for example, information corresponding to mass, firmness and softness, reaction force, surface pattern, small shapes, surface softness and roughness, and a sense of cold).

As described above, the haptic unit 370 is applied to command confirmation and control through tactile feedback, various sensational games, navigation, various messages and communication, and the mobile device for the visually impaired. It can provide an interaction function.

In addition, the haptic unit 370 may greatly increase the realism of a remote environment or a virtual environment through the transmission of touch, and provide a function of touching and buying a product in an online shopping mall that was only visible in a network-based environment. In addition, it can provide an online museum function that can touch the actual artifacts, and in the text-based messenger environment, it provides a function of conveying emotions through the touch, such as shaking hands and tapping shoulders. It may provide an interactive function between the terminal and the user.

The fragrance generation unit (or olfactory interface) 380 is configured on one side (or one surface) of the digital signage 300.

The human sense of smell senses the smell of the surrounding environment and accepts information about the environment. The smell of baking, burning, and the smell of a familiar lover are being informed to us. As such, the scent may be used in the signage system 10 to convey representative information about the condition, situation, or event.

In addition, the olfactory interface is divided into an odor recognition device (not shown) that recognizes the smell of the surrounding environment and the scent generator 380 that outputs a desired scent around the user.

Odor recognition devices, commonly referred to as electronic noses, are machines that replace a series of processes, such as the brain processing the information of odors detected by humans through the nose. The electronic nose is composed of a sensor (not shown) corresponding to the olfactory cells of the human nose and a computer or microprocessor (not shown) corresponding to the human brain. Just as a person processes odor information detected by the nose's olfactory cells in the brain to recognize odors, when a sensor responds to an odor molecule, the microprocessor discriminates the odor by pattern recognition software about how the sensor responds.

The basis of the electronic nose is to use several different kinds of chemical gas sensors in an array. Data measured by the sensor array (sensor array) has a characteristic that can be distinguished according to the smell, because each sensor is made of a material having a different sensitivity depending on the chemical composition of the smell.

In addition, the fragrance generation unit 380 is associated with the MR content being displayed while displaying the MR content corresponding to the current time on the display unit 350 under the control of the playback schedule information under the control of the control unit 390. If there is a predetermined smell information, the fragrance (or fragrance) is sprayed (or output / generated / discharge) to the outside according to the corresponding olfactory information (including rose fragrance, perfume fragrance, burnt fragrance of food, etc.) .

In addition, the fragrance sprayed (or generated / output) by the fragrance generating unit 380 is recognized through the electronic nose, it is learned by the control of the control unit 390 to further refine the fragrance later It may also be used to dispense (or output / generate / generate).

In addition, the scent generator 380 may be applied to a technology associated with an online game.

In other words, the fragrance generating unit 380 may increase the sensibility of the game by outputting a gunpowder smell or a burning smell of an alien creature or a rubber burning smell for a racing game in an online game. Game players today demanding vast amounts of material and the best gameplay environment are highly competitive, so if the fragrances are delivered to existing game content and are advantageous for their activities, fragrance technology can be applied to everyday life. This suggests that fragrance applications can begin in certain market segments with high revenue potential, as shown in the example of CD-quality music and high-definition graphics.

In addition, the fragrance generation unit 380 may increase the memory and learning ability for the smell by using the fragrance in the personal space.

In addition, the fragrance generating unit 380 may apply the fragrance as an information display medium that is very useful for the visually impaired, the hearing impaired, and other individuals with special needs. High-quality computer-controlled sound devices, tactile research, and inexpensive computer hardware and software will provide an enhanced way for individuals to use their senses to communicate and receive information from the outside world.

In the exemplary embodiment of the present invention, the haptic unit 370 and / or the fragrance generating unit 380 are described as being included in some of the components of the digital signage 300 to operate, but the present invention is not limited thereto. The haptic unit 370 and / or the scent generating unit 380 operate in a stand alone form, and when there is MR content being output from the digital signage 300, the haptic unit 370. And / or the MR content (or attribute information of the marker included in the MR content) transmitted from the digital signage 300 in the fragrance generation unit 380, and the marker included in the received MR content. It may operate in response to auditory information and / or olfactory information based on the attribute information of.

That is, the MR in the digital signage 300 in a state where the haptic part 370 and / or the scent generator 380 are independently configured and disposed (or installed) adjacent to the digital signage 300. When the content (or AR content / VR content / other format content) is being output, the haptic unit 370 and / or the scent generating unit 380 may transmit MR content (or, transmitted from the digital signage 300). Attribute information of the marker included in the corresponding MR content).

The haptic unit 370 and / or the scent generator 380 may generate vibrations corresponding to the auditory information or correspond to the olfactory information based on the attribute information of the marker included in the received MR content. Spray the fragrance.

The controller 390 executes an overall control function of the digital signage 300.

In addition, the controller 390 executes an overall control function of the digital signage 300 using the programs and data stored in the storage 320. The controller 390 may include a RAM, a ROM, a CPU, a GPU, a bus, and the RAM, a ROM, a CPU, a GPU, and the like may be connected to each other through a bus. The CPU may access the storage 320 to boot using the O / S stored in the storage 320 and use various programs, contents, data, etc. stored in the storage 320. Various operations can be performed.

In addition, the controller 390 displays the MR content (or an advertisement or promotion content separate from the MR content) provided from the server 200 based on the received playback schedule information and the current time. The voice output unit 350 outputs the voice output unit 360, the haptic unit 370, and the scent generator 380.

That is, the controller 390 is included in the MR content provided from the server 200 based on the received play schedule information and the current time (or including advertisement or promotion content separate from the MR content). Image information is displayed (or output) through the display unit 350. In this case, the controller 390 displays the MR content transmitted from the server 200 in real time through the display unit 350 or from the server 200 in consideration of the storage capacity of the storage unit 320. After temporarily storing the transmitted MR content in the storage unit 320, the MR content temporarily stored in the storage unit 320 may be displayed through the display unit 350.

In addition, the controller 390 displays (or displays) the MR content on the display unit 350 in the digital signage 300 based on the attribute information of the marker included in the MR content. In addition to visual information related to the content, audio information, tactile information, olfactory information, and the like are provided (or output).

That is, the controller 390 displays the MR content corresponding to the current time on the display unit 350 according to the play schedule information, and when there is auditory information preset with respect to the corresponding MR content being displayed, Auditory information (or sound / voice information corresponding to the auditory information) is output through the voice output unit 360.

In addition, the controller 390 displays the MR content corresponding to the current time through the display unit 350 according to the play schedule information, and when there is preset tactile information related to the corresponding MR content being displayed, Tactile information (or vibration information corresponding to the tactile information) is output through the haptic unit 370.

In addition, the controller 390 displays MR content corresponding to a current time according to the play schedule information through the display unit 350, and when there is preset olfactory information related to the corresponding MR content being displayed, The fragrance (or fragrance) corresponding to the olfactory information is sprayed (or output) through the fragrance generation unit 380.

In addition, when there is at least one other terminal 400 located within a preset radius from the digital signage 300, the controller 390 is the at least one other terminal 400 through the communication unit 310. ). In this case, the communication connection method between the digital signage 300 and the other terminal 400 may be a wireless communication method.

In addition, the control unit 390 is connected to the other terminal 400 via the communication unit 310 to the MR content that is being displayed on the display unit 350 (or including advertising or promotion content separate from the MR content) through the communication unit 310 To be sent).

As such, the controller 390 may output the content and additional information related to the content (for example, auditory information, tactile information, olfactory information, etc.) through the digital signage 300, or the digital signal. Information (for example, the content and additional information related to the content) being output through the digital signage 300 may be provided to at least one other terminal 400 adjacent to the signage 300. .

In addition, according to a voice and / or an operation of a specific user adjacent to the digital signage 300 recognized through the sensor unit 340, the MR content (or content) displayed on the display unit 350 may be displayed. When a marker is selected, the controller 390 provides an additional function related to the marker.

That is, when a marker displayed on one side of the MR content being displayed on the display unit 350 is selected according to the recognized voice and / or operation, the controller 390 may receive another terminal 400 to receive the MR content. ) Displays a screen (or user interface / UI) for receiving user information on the display unit 350.

In addition, when receiving user information about another terminal 400 to receive the MR content according to a user input (or user selection / touch), the controller 390 transmits the received information to another terminal 400. The MR content being displayed on the display unit 350 is provided to another terminal 400 corresponding to user information about another terminal 400 through the communication unit 310 based on the user information. send. In this case, the user information for the another terminal 400 includes a phone number / identification information, an e-mail address, SNS account information, etc. of another terminal 400.

In addition, when a specific menu displayed on the other side of the MR content displayed on the display unit 350 is selected according to the recognized voice and / or operation, the controller 390 may control the specific user and / or the server ( In conjunction with the 200, information (or content) corresponding to the selected specific menu is output through the display unit 350, the voice output unit 360, the haptic unit 370, and the scent generating unit 380. Or perform a function corresponding to the selected specific menu and output a result of performing the corresponding function.

In the embodiment of the present invention, it is described that the various information is shared through the interaction between the digital signage 300 and the specific user and the another terminal 400, but is not limited to this, between the digital The niche 300 receives user information on the another terminal 400 according to the operation of the specific user and / or a voice recognition result, user information on the received another terminal 400, and is currently being output. Information about an MR content (or arbitrary content), etc. is transmitted to the server 200, and user information about another terminal 400 transmitted from the digital signage 300 at the server 200, present The MR content being output from the digital signage 300 may be transmitted to the corresponding another terminal 400 based on information on the MR content (or arbitrary content) being output.

In addition, the digital signage 300 outputs specific content according to the current time based on the reproduction scheduling information managed by the server 200, but the interaction between the digital signage 300 and the specific user is performed in this way. If so, the output of the specific content to be output at the current time is temporarily suspended according to the reproduction scheduling information, and the interaction function between the digital signage 300 and the specific user may be preferentially processed.

In addition, after the interaction between the digital signage 300 and the specific user is completed, the digital signage 300 displays specific content corresponding to the current time based on the playback scheduling information and the time information of the current time. You can print

The importance of interfaces is growing as computer technology advances. The interface between the computer and the interface is inseparable, and in order for a person to use the computer conveniently, the interface of the computer has to rely on the five senses of the human being. Accordingly, in the embodiment of the present invention, the digital signage 300 applies a multimodal interface technology that utilizes human's intuitive operation, tactile sense and sense of smell, and fuses multi-modality to the interface technology that depends only on audiovisual. .

Here, the dictionary meaning of the multi-modal is 'to include or have multiple modes, aspects, and senses'. The multimodal interface may allow a human and a computer to interact by using two or more input / output modalities.

For the multi-modal interface, voice recognition, 2D / 3D gesture recognition, lip movement recognition, pupil movement recognition, etc., as well as input technology for modality that can be input by simple direct manipulation such as keyboard, mouse, pen, touch screen, etc. The same input techniques for more complex and advanced modalities are used. In addition, modality technologies such as speech synthesis, 2D / 3D graphics, and braille output are used as output technologies. Convergence technology to fuse these individual modalities with each other and provide a means for inputting / outputting the modalities usable in a system may be an important interface of next-generation computers with emphasis on portability and mobility.

In addition, the multimodal interface is a human-centered interface that allows a dwelling person to interact with a computer in a similar manner. It provides the advantage of selecting different input / output method for environment with limited specific interface or disabled person. It is also possible to get reliable input / output result by fusing two or more inputs.

The multimodal system uses various input modes such as sound, voice, handwriting, keyboard, and the output may be configured with one or more multiple output modes such as voice, text, graphics, sound, and video.

In addition, in a multimodal framework configuration, a user enters into the system and views or hears information represented by the system. The Interaction Manager coordinates the data between the various I / O components appropriately, manages the state and contextual information that each component runs on, and the Session component provides the Interaction Manager to provide session and state management for multimodal applications. Provides an interface with Data from various input devices is converted to a generalized format and passed to the interaction manager.

In addition, in order to generalize the data of these input devices, the W3C uses the EMMA Markup Language (Extensible Multimodal Annotation Markup Language) format, and the InkML markup language for digital ink data. Here, the EMMA is a markup language for expressing various input results such as voice, text, GUI, ink, etc. in a multi-modal interaction framework in a meaningful data format, and the InkML is an ink pen inputted using an electronic pen or a stylus. Markup language for providing the data format to be represented.

As such, the digital signage 300 is implemented through various markup languages (including EMMA markup language, InkML markup language, Speech Application Language Tags (SALT), etc.) to enhance the user's experience. It can provide an interface that can increase reliability when trying to access any application. In addition, a user can interact with one device or modalities at the same time, and even more devices or modalities at the same time. That is, the application may be extended in various forms of interaction such as pen input, multimedia output, video input, haptic input and output, and TTY communication.

The other terminal 400 communicates with the server 200, the digital signage 300, and the like.

In addition, when the other terminal 400 is located within the communication radius of the digital signage 300, the other terminal 400 is output from the digital signage 300 transmitted from the digital signage 300. Receives the MR content (or a separate advertisement or promotion content produced in a different format from the MR content).

In addition, the MR content being output from the digital signage 300 by the interaction between the specific user and the digital signage 300 (or a separate advertisement or content produced in a format different from the MR content) If the other terminal 400 is selected (or selected) as a terminal to be provided, the other terminal 400 is the digital signage 300 and / or the digital transmitted from the server 200 Receives MR content (or a separate advertisement or promotional content produced in a format different from MR content) output from the signage 300.

In addition, the other terminal 400 displays the received MR content (or image information included in MR content included in a separate advertisement or promotion content produced in a format different from MR content). Here, the other terminal 400 displays the MR content, outputs voice information based on auditory information related to the MR content, and is provided to the other terminal 400 based on tactile information related to the MR content. A vibration corresponding to the tactile information may be output through the vibrator (not shown).

In addition, the another terminal 400 may display additional MR content (or additional animation information) related to the marker according to a user's selection of the displayed MR content, and a direct purchase screen for the 3D object related to the marker. (Or a function of displaying a direct purchase function) and a sharing function to any specific terminal (not shown) may be performed.

That is, when a marker displayed on one side of MR content displayed on the another terminal 400 is selected, the another terminal 400 displays additional MR content (or additional animation information) related to the corresponding marker.

As described above, various information may be shared in real time through the interaction between the digital signage 300 and the another terminal 400.

In addition, the other terminal 400 may detail the object related to the object included in the additional MR content on one side where the additional MR content is displayed (or by overlaying / applying augmented reality function on one side where the additional MR content is displayed). Display information.

In this case, when the marker displayed on one side of the MR content displayed on the other terminal 400 is selected, the detailed information related to the object is between the server 200 and / or the digital in real time in the other terminal 400. After requesting the detailed information related to the object to the niger 300, and in response to the detailed information related to the object in real time from the server 200 and / or the digital signage 300, the other terminal It may be information provided to the 400.

Here, the detailed information includes a product name, a product unique model number, a manufacturing date, a manufacturer, code information according to a product type, a country of origin, a price, a weight, and promotion information (or event information) related to the object.

In addition, when the direct purchase menu (or item) related to the marker displayed on the other side of the MR content displayed on the other terminal 400 is selected, the other terminal 400 directly purchases the 3D object related to the marker. Display the screen.

That is, when the immediate purchase menu is selected with respect to the marker displayed on the other side of the MR content displayed on the other terminal 400, the other terminal 400 performs a function of purchasing a product corresponding to the 3D object related to the marker. Displays a screen (for example, a shopping cart screen, a payment screen, etc.).

As such, the other terminal 400 provides a one-stop service for product search (or product search / recommendation) and purchase using MR content including the 3D object and a marker associated with the 3D object. By providing a user's convenience can be improved.

In addition, when a shared item displayed on another side of the MR content displayed on the other terminal 400 is selected, the other terminal 400 may receive any MR content according to a user selection (or a user touch). Select (or select) a terminal. In addition, the other terminal 400 provides (or transmits) MR content including the marker and the 3D object to the selected arbitrary terminal. In this case, the other terminal 400 transmits MR content including the marker and the 3D object to a social network service (SNS) related to the user of the other terminal 400 or the user of the selected arbitrary terminal. (Or share / provide).

The terminal 100 and the other terminal 400 according to an embodiment of the present invention are a communication unit, a storage unit, a camera unit, a sensor unit, a display unit, a voice output unit, a haptic unit included in the digital signage 300, It may be configured to include at least one or more components of the components, such as the fragrance generating unit, the control unit.

As described above, MR contents are created by adding one or more markers based on 3D objects generated through a dedicated tool or 3D objects through object recognition in an image captured by a camera, and visually, It may be provided through a signage device related to hearing, smell, hearing, and the like.

Hereinafter, a signage method for providing MR content including a 3D object and a marker according to the present invention will be described in detail with reference to FIGS. 1 to 9.

4 is a flowchart illustrating a signage method for providing MR content including 3D objects and markers according to an embodiment of the present invention.

First, the terminal 100 uses a dedicated tool (or a dedicated program) or image information including an object photographed through a camera unit (not shown) provided in the terminal 100 (or a corresponding 3D object). 3D object data related to the 3D object) is generated (or manufactured / modeled).

That is, the terminal 100 generates (or fabricates / models) a 3D object related to a specific object by using a dedicated tool (or dedicated program).

In addition, the terminal 100 photographs an image including an arbitrary object through a camera unit included in the terminal 100, recognizes an object in the photographed image (or image information), and then the terminal 100. The 3D object is generated (or manufactured / modeled) using the recognized object through a 3D object generation program (not shown) installed in advance. In this case, the image may be provided from any terminal (not shown) or server 200 communicating with the terminal 100. In this case, the generated (or produced / modeled) 3D object is generated to include visual information, auditory information, tactile information, olfactory information, etc. related to the 3D object, and thus provides visual and auditory information when the information on the 3D object is provided. It can be configured to provide information about the sense of touch, sense of smell, etc. at the same time. In this case, the visual information includes shape information, color information, material information, pattern information, design information, etc. of the corresponding 3D object. The auditory information may include sound information related to the corresponding 3D object. In addition, the tactile information may include information on muscle sensation related to the 3D object (for example, information on mass, firmness, and reaction force related to the 3D object), and information on texture related to the 3D object (eg For example, the surface pattern related to the 3D object, small shapes, including information about the smoothness and roughness of the surface, the sense of warmth and the like). In addition, the olfactory information includes fragrance (or fragrance) information related to the 3D object.

For example, as shown in FIG. 5, the first terminal 100 manufactures the 3D wireless vacuum cleaner 510 using a dedicated tool pre-installed in the first terminal. Here, the manufactured 3D wireless vacuum cleaner may include first visual information including shape information, color information, material information, pattern information, design information, and an operation mode of the corresponding wireless vacuum cleaner (for example, the first stage, First auditory information including noise information according to an operation mode of a second stage, a third stage, and the like, first touch information including magnitude information of vibration according to an operation mode of the wireless vacuum cleaner, and a corresponding wireless vacuum. The first olfactory information includes information on fragrances emitted to the outside in predetermined units of time during operation of the cleaner (S410).

Thereafter, a preset marker insertion button (or menu / item) displayed on one side of the screen of the terminal 100 in order to insert a marker related to the generated (or manufactured) 3D object (or the terminal 100). When a marker insertion button provided at one side is selected, the terminal 100 arranges a marker (or an object marker) adjacent to the generated 3D object.

In addition, after the preset marker insertion button displayed on one side of the screen of the terminal 100 is selected (or dragged) using the drag and drop function, the screen is displayed on the screen of the terminal 100 by the drag and drop function. When dropped over a 3D object being displayed (or when dropped to an area / position within a preset radius from the 3D object), the terminal 100 may move to any position (or a specific position) spaced apart from the 3D object by a preset distance. The marker may also be arranged in).

In this case, the terminal 100 is arranged to be spaced apart by a predetermined interval with respect to the 3D object, and the marker is set to maintain a constant interval on the 3D object even if the 3D object is rotated or reduced / enlarged in three dimensions (Or layout / design).

That is, the terminal 100 configures a layer in which the 3D object operates and a layer in which the marker operates (or displays) differently, so that the marker may be rotated, reduced, enlarged, or moved in the 3D object. It can be set to maintain the distance from the object by a certain interval.

For example, as shown in FIG. 5, after the 3D type wireless vacuum cleaner 510 is manufactured, the marker insertion menu 520 displayed on one side of the screen of FIG. 5 may be selected (or touched). 6, the first terminal arranges the first marker 620 at a position separated by a predetermined interval from the 3D type wireless vacuum cleaner 610. At this time, when the 3D-type wireless vacuum cleaner 610 shown in FIG. 6 is enlarged by user control, as shown in FIG. 7, the first terminal is the enlarged 3D-type wireless vacuum cleaner 710. The first marker 720 may be disposed at a position spaced apart from each other by a predetermined interval so that a predetermined interval is maintained between the 3D wireless vacuum cleaner 710 and the first marker 720 (S420). ).

Subsequently, when the marker displayed on the terminal 100 is selected (or touched / double-clicked), the terminal 100 displays attribute information on the selected marker based on a user input (or user touch / user selected). Set it. Here, the attribute information (or the attribute information of the marker) for the marker may include unique code information of an object (or 3D object) related to the marker, unique code information of the marker related to the object, and at least one MR content related to the object. Information about (or address information / link address where at least one MR content related to the object is stored), visual information, auditory information, tactile information, olfactory information, etc., related to the 3D object corresponding to the marker. In this case, the at least one MR content related to the object may be MR content corresponding to at least one other 3D object having the same code or similar code with respect to the object.

Here, when setting the information on the at least one MR content related to the object, the terminal 100 is associated with the user information (eg, preferences, purchases) related to the user (or a specific / arbitrary user) of the terminal 100 Historical information, information about the products the user most searched for in relation to that object, information on the products sold most in relation to that object, promotional information, and big data in relation to that object and / or user). Based on this, at least one MR content related to the object may be selected (or set) from among a plurality of MR contents stored in the terminal 100 and / or the server 200 in advance.

As such, the terminal 100 accumulates the user information to provide the optimized MR content to the corresponding user (or a specific / arbitrary user), and the user 100 through machine learning on the accumulated user information. The attribute information for the selected marker of the adaptive conversion form may be set by reflecting the related information.

For example, first detailed information (or first product information) related to the wireless vacuum cleaner (for example, a product name, a product unique model number, a manufacturing date, a manufacturer, and code information according to a product type) is registered in advance. In the (or input) state, when the first marker 620 is selected on the screen in which the 3D type wireless vacuum cleaner 610 and the first marker 620 shown in FIG. 6 are displayed, the first marker 620 is selected. The first terminal checks first detailed information corresponding to the 3D type wireless vacuum cleaner among the plurality of detailed information stored in advance.

The first terminal may include first visual information including shape information, color information, material information, pattern information, design information, and the like associated with the 3D type wireless vacuum cleaner set when the 3D type wireless vacuum cleaner is manufactured. First auditory information including noise information according to an operation mode of the wireless vacuum cleaner (for example, operation modes such as 1st stage, 2nd stage, and 3rd stage), and magnitude of vibration according to the operation mode of the wireless vacuum cleaner. First tactile information including information and the like, and first olfactory information including information on fragrances emitted to the outside in predetermined units of time when the wireless vacuum cleaner operates.

In addition, the first terminal has a second MR content having the same code or a similar code as the 3D object related to the wireless vacuum cleaner among a plurality of MR content stored in advance (for example, a wireline reflecting the user's preference according to machine learning) The second address information and the third address information in which the MR content related to the vacuum cleaner and the third MR content (for example, the MR content related to another wireless vacuum cleaner most sold in the last month) are stored. Here, the second MR content includes a 3D-type wired vacuum cleaner and a marker related to the 3D-type wired vacuum cleaner, and the third MR content includes another 3D-type wireless vacuum cleaner and another 3D-type wireless vacuum. And a marker associated with the cleaner.

The first terminal may further include first detailed information, first visual information, first auditory information, first tactile information, first olfactory information, and second MR content related to the wireless vacuum cleaner. And set first attribute information on the first marker including second address information, third address information, and the like, in which the third MR content is stored (S430).

Subsequently, when a preset preview menu (or button / item) displayed on one side of the terminal 100 is selected, the terminal 100 displays a result image of MR content creation as a preview screen.

That is, when the preview menu is selected, the terminal 100 displays a result image of the MR content creation (or an image including the generated object and a marker disposed adjacent to the object) in a preview form. . In this case, the resultant image of the MR content authoring may include an immediate purchase menu for providing a purchase function related to the object, a sharing menu for providing a sharing function for the MR content, and the like.

For example, when the preset preview menu 630 is selected on the screen on which the 3D type wireless vacuum cleaner 610 and the first marker 620 shown in FIG. 6 are displayed, as shown in FIG. 8. As described above, the first terminal displays the result image 800 of the MR content creation corresponding to the 3D wireless vacuum cleaner 810, the first marker 820 related to the wireless vacuum cleaner, etc., respectively, in a pop-up form. In this case, the MR image authoring image 800 includes a direct purchase menu 830 for providing a purchase function for the wireless vacuum cleaner 810, a 3D wireless vacuum cleaner 810, and a first marker. In operation S440, a sharing menu 840 for sharing the MR content including the content 820 may be included.

Subsequently, when a preset storage menu (or completion menu / button / item) displayed on the other side of the terminal 100 is selected, the terminal 100 is a 3D object, a marker disposed adjacent to the 3D object. Create MR content including the like. In this case, the MR content is a corresponding 3D object, property information (or product information) of the 3D object, code associated with the 3D object, unique identification information of the code, type of code, marker corresponding to the 3D object, marker location information, Property information of a marker (including, for example, information on at least one MR content related to the corresponding 3D object, visual information related to the 3D object corresponding to the marker, auditory information, tactile information, olfactory information, etc.) Immediately include purchase information, shared information and more.

That is, when the preset storage menu is selected, the terminal 100 checks whether the attribute information setting of the marker according to the user selection is completed for the marker.

In addition, when the check result, the property information setting for the marker is not completed, the terminal 100 indicates that the property information setting of the marker is not completed, and the property information setting of the marker The guide information is displayed on one side.

In addition, as a result of the checking, when the attribute information setting for the marker is completed, the terminal 100 generates MR content including the 3D object, a marker related to the 3D object, and the like.

In this case, when the preset storage button is selected while a video is generated using the generated 3D object, the terminal 100 identifies unique identification information of a code related to the 3D object and a marker corresponding to the 3D object. (Or match / interlate / update) each of the markers so as to be adjacent to the 3D object included in the video, based on (or unique identification information of the code related to the area / location where the marker is placed). MR content in the form can be generated.

In addition, the terminal 100 transmits the generated MR content, identification information of the terminal 100, etc. to the server 200 in order to share the generated MR content. Here, the identification information of the terminal 100 includes MDN, mobile IP, mobile MAC, Sim (subscriber identification module) card unique information, serial number and the like.

For example, when a preset storage menu 640 is selected on the screen on which the 3D type wireless vacuum cleaner 610 and the first marker 620 shown in FIG. 6 are displayed, the first terminal is configured to display the 3D. Wireless vacuum cleaner 610 of the form, the first marker 620 disposed adjacent to the wireless vacuum cleaner 610, location information of the first marker 620, attribute information of the first marker 620, etc. Create a first MR content comprising a. In this case, the attribute information of the first marker may include first detailed information corresponding to the wireless vacuum cleaner, a second MR content having a same / similar code as a 3D object related to the wireless vacuum cleaner, and a second stored MR content. Address information to third address information, first visual information related to the wireless vacuum cleaner, first auditory information, first tactile information, first olfactory information, and the like are included (S450).

Thereafter, the server 200 receives MR content transmitted from the terminal 100, identification information of the terminal 100, and the like.

In addition, the server 200 stores (or manages) a plurality of MR contents respectively transmitted from the plurality of terminals 100.

In addition, the server 200 is a specific digital based on a plurality of MR content stored in the server 200 (or including a separate advertising or promotional content produced in a format (or form) different from the MR content) Scheduling of the content to be output from the signage 300 is performed.

That is, the server 200 generates playback scheduling information including information on content to be output by date and / or time zone according to time cells for a plurality of MR contents.

For example, the server 200 receives first MR content transmitted from the first terminal.

In addition, the server 200 is controlled by the administrator of the server 200 in the first digital signage 300, January 1, 2018 0: 0: 0 seconds ~ January 7, 2018 24 hours 0 The first reproduction scheduling information for one week until minute 0 seconds is generated (S460).

Thereafter, the server 200 corresponds to the generated reproduction schedule information and a plurality of MR contents (or separate advertisements or promotion contents produced in a format different from the MR contents) according to the reproduction schedule information. The digital signal is transmitted to the digital signage 300 related to the reproduction schedule information. At this time, the server 200 transmits the generated playback schedule information to the digital signage 300 in advance, and sequentially outputs a plurality of MR contents to be output from the corresponding digital signage 300 according to the playback schedule information. By transmitting to the digital signage 300, even if the digital signage 300 does not have a large storage capacity, a plurality of MR contents provided from the server 200 may be sequentially output.

For example, the server 200 may include the generated first play schedule information (eg, play schedule information about first MR content, eleventh MR content, twelfth MR content, etc.) and the first play schedule information. In operation S470, the first digital signage is transmitted.

Subsequently, the digital signage 300 may include a plurality of MR contents (or separate advertisements or promotions produced in a format different from MR contents) according to the play schedule information transmitted from the server 200 and the play schedule information. Content, etc.).

In addition, the digital signage 300 includes MR content provided from the server 200 based on the received playback schedule information and the current time (or content for advertisement or promotion separate from MR content / MR, etc.) Video information included in the content). In this case, the digital signage 300 outputs MR content transmitted from the server 200 in real time, or MR content transmitted from the server 200 in consideration of the storage capacity of the digital signage 300. After temporarily storing, the temporarily stored MR content may be output.

In addition, the digital signage 300 displays (or displays) the MR content (or image information included in the MR content) in the digital signage 300, and attributes of the markers included in the MR content. Based on the information, audio information, tactile information, olfactory information, etc. as well as visual information related to the MR content are provided (or output).

That is, the digital signage 300 displays MR content corresponding to the current time according to the play schedule information, and outputs the auditory information when there is preset auditory information related to the corresponding MR content being displayed. .

In addition, the digital signage 300 displays MR content corresponding to the current time according to the playback schedule information, and when there is tactile information related to the corresponding MR content being displayed, the digital signage 300 is displayed on the digital signage 300. The haptic unit 370 outputs the tactile information.

In addition, the digital signage 300 displays MR content corresponding to a current time according to the playback schedule information, and when there is olfactory information related to the corresponding MR content being displayed, the digital signage 300 is displayed on the digital signage 300. The fragrance (or fragrance) is sprayed (or output) to the outside through the included fragrance generating unit 380.

In this case, when there is at least one other terminal 400 located within a preset radius from the digital signage 300, the digital signage 300 is a wireless communication scheme with the at least one other terminal 400. By connecting to, at least one other terminal 400 transmits the MR content (or content, such as advertising or promotional content separate from the MR content) being output from the digital signage 300. In addition, the other terminal 400 receives the MR content transmitted from the digital signage 300 (or includes a separate advertisement or promotion content produced in a different format from the MR content), and receives the received content. The displayed MR content (or a separate advertisement or promotional content produced in a different format from the MR content). Here, the other terminal 400 displays the MR content, outputs voice information based on auditory information related to the MR content, and is provided to the other terminal 400 based on tactile information related to the MR content. A vibration corresponding to the tactile information may be output through the vibrating unit (not shown).

For example, the first digital signage displays the first MR content related to the wireless vacuum cleaner based on the first playback schedule information transmitted from the server 200 and the current time, and displays the first MR content and the first MR content. Outputs noise information while the wireless vacuum cleaner is operating in relation to the related voice information and the first MR content, and transmits the wireless information through the first haptic unit 370 provided on one side of the first digital signage. Outputs a vibration corresponding to the magnitude information of the vibration according to the operation mode of the vacuum cleaner, and a predetermined time unit (for example, 1 minute) through the first fragrance generating unit 380 provided on one side of the first digital signage Spray a predetermined amount of perfume.

At this time, when the second terminal to the fourth terminal is located within a predetermined radius (for example, 2 meters) from the first digital signage, the first digital signage is a wireless communication scheme such as Bluetooth, Zigbee, Wi-Fi, etc. The second terminal to the fourth terminal to communicate with each other, and transmits the first MR content and the like output from the first digital signage to the second terminal to the fourth terminal, respectively.

In addition, each of the second terminal to the fourth terminal receives the first MR content, etc. being output from the first digital signage transmitted from the first digital signage, respectively, and outputs the received first MR content. do. In this case, each of the second terminal to the fourth terminal includes information related to the first MR content and the first MR content (for example, auditory information and tactile sense) through a display unit, a voice output unit, and a vibration unit provided in the corresponding terminal. Information, etc.) can be output (S480).

Thereafter, the digital signage 300 recognizes a voice and / or an operation of a specific user adjacent to the digital signage 300.

In addition, when a marker displayed on one side of the MR content being displayed in the digital signage 300 is selected according to the recognized voice and / or operation, the digital signage 300 provides additional functions related to the marker. to provide.

That is, when a marker displayed on one side of the MR content being displayed in the digital signage 300 is selected according to the recognized voice and / or operation, the digital signage 300 may receive the MR content. Displaying a screen (or user interface / UI) for receiving user information on the other terminal 400, for another terminal 400 to receive the MR content according to the user input (or user selection / touch) Receive user information. In this case, the user information about the another terminal 400 includes a phone number / identification information, an e-mail address, SNS account information, etc. of another terminal 400.

In addition, the digital signage 300 is connected between the digital to another terminal 400 corresponding to the user information for another terminal 400 based on the received user information for another terminal 400 The MR content, etc. being displayed on the display 300 is provided (or transmitted).

In this case, the another terminal 400 receives the MR content provided from the digital signage 300 and outputs the received MR content.

Also, the another terminal 40 may display additional MR content (or additional animation information) related to the marker according to a user's selection of the displayed MR content, and a direct purchase screen for the 3D object related to the marker. (Or a function of displaying a direct purchase function) and a sharing function to any specific terminal (not shown) may be performed.

As described above, various information may be shared in real time through the interaction between the digital signage 300 and the another terminal 400.

For example, the first digital signage recognizes an operation (or motion) of a specific user adjacent to the first digital signage and is displayed on the first digital signage based on the recognized specific user's motion. 1 When the first marker displayed on one side of the MR content is selected, as shown in FIG. 9, a screen 900 for receiving input of eleventh user information to be provided with the first MR content displayed in the first digital signage. ). The first digital signage may be in a state where a plurality of displays are arranged in a 4 × 4 form.

Further, the first digital signage recognizes the action and / or voice of the specific user, and based on the recognized action and / or voice, the eleventh user information (e.g., an eleventh e-mail relating to the specific user). Address (or address).

In addition, the first digital signage transmits the eleventh MR content being output from the first digital signage to the eleventh e-mail address associated with the specific user based on the received (or input) eleventh user information. .

In addition, the specific user may access an account related to an eleventh e-mail address related to the specific user, and may check (or output) the eleventh MR content transmitted from the first digital signage (S490).

As described above, an MR content may be created by adding one or more markers based on a 3D object generated through a dedicated tool or a 3D object through object recognition in an image captured by a camera, as described above. In addition, the produced MR content is provided through a signage device related to visual, auditory, olfactory, auditory, etc. to provide various additional information regarding 3D objects, and to provide a virtual experience and a simulation function of a product to improve user satisfaction. It is possible to increase the use of MR content according to the user's demand.

The above description may be modified and modified by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

The present invention creates MR content by adding one or more markers based on the 3D object generated through a dedicated tool or 3D object through object recognition in an image captured by a camera, and visualizes the produced MR content. By providing through signage devices related to hearing, smell, hearing, etc., it provides various additional information related to 3D objects, improves user satisfaction by providing virtual experience and simulation of products, and MR contents according to user needs. It can be widely used in virtual reality field, augmented reality field, mixed reality field, terminal field, panoramic image field, shopping information provision field, marketing field, server field, multi-modality field, etc. have.

Claims (1)

In a signage method for providing MR content including 3D objects and markers,
Outputting, by the digital signage, the MR content provided from the server based on the reproduction schedule information and the current time information;
Recognizing, by the digital signage, a voice of a user; And
And performing a function corresponding to the selected marker when the marker included in the output MR content is selected by the digital signage based on the recognized user's voice. Signage method for providing MR content.

Images