KR20180004643A - Terminal and control method thereof - Google Patents

Abstract

The present invention discloses a terminal and a control method thereof. That is, according to the present invention, a panoramic image of 2D type is photographed. An object is recognized in the photographed panoramic image. An additional object of 3D type related to the recognized object is inserted to generate an augmented reality image. Additional real-time information related to the additional object is displayed when the additional object of 3D type inserted in the augmented reality image is selected. Therefore, it is possible to provide a user interface for user convenience and to improve the convenience of a user by providing additional information related to an object which interests the user on a current screen.

Description

[0001] The present invention relates to a terminal and a control method thereof,

In particular, the present invention relates to a terminal and a control method thereof, and more particularly to a terminal and a control method thereof, in which a 2D panoramic image is taken, an object is recognized in the panoramic image, Real time additional information related to the additional object when the additional 3D object is selected in the augmented reality image, and a control method thereof.

2. Description of the Related Art Generally, a mobile terminal is a device that performs a Global Positioning System (GPS) function and a call function (or a call function) and provides a result of the operation to a user.

These mobile terminals are able to transmit and receive more information in a shorter time and have various functions, but there is a lack of UI (User Interface) for user-oriented convenience in accordance with these functions.

Korean Patent Laid-Open No. 10-2013-0084274 [entitled " 3D virtual reality Internet shopping mall]

An object of the present invention is to provide a terminal that captures a 2D panoramic image, recognizes an object in the captured panoramic image, and inserts an additional 3D object in association with the recognized object to generate an augmented reality image, Method.

Another object of the present invention is to provide a terminal and a control method thereof for displaying real-time additional information related to the additional object when an additional object of 3D form inserted into the augmented reality image is selected.

A method of controlling a terminal according to an exemplary embodiment of the present invention includes: capturing a two-dimensional image by a camera; Displaying the photographed image by a display unit; Generating augmented reality image by inserting a three-dimensional additional object into the photographed two-dimensional image by the control unit; And displaying the generated augmented reality image by the display unit, wherein the additional object is configured in a three-dimensional form, and the real-time supplementary information related to the object included in the photographed image, Or may be an object for providing information.

The present invention relates to a method for capturing a 2D panoramic image, recognizing an object in the captured panoramic image, and inserting an additional 3D object in association with the recognized object to generate an augmented reality image, It is possible to facilitate information provision.

In addition, when the 3D object of the 3D form embedded in the augmented reality image is selected, real-time additional information related to the additional object is displayed, thereby providing a user interface for user's convenience. There is an effect that the convenience of the user by providing additional information related to a visible object can be improved.

1 is a block diagram showing the configuration of a guide apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a UE according to an embodiment of the present invention.
3 is a diagram illustrating an example of an exhibition product according to an embodiment of the present invention.
4 is a diagram illustrating an example of POP according to an embodiment of the present invention.
5 is a view showing an example of a front end according to an embodiment of the present invention.
6 is a diagram illustrating an example of a banner advertisement according to an embodiment of the present invention.
7 is a diagram illustrating an example of a poster advertisement according to an embodiment of the present invention.
8 is a view showing an example of a signboard according to an embodiment of the present invention.
9 is a flowchart illustrating a method of controlling a terminal according to a first embodiment of the present invention.
10 to 13 are diagrams illustrating screens of a terminal according to a first embodiment of the present invention.
Fig. 14 6 is a flowchart illustrating a method of controlling a terminal according to a second embodiment of the present invention.
15 to 21 are diagrams illustrating screens of a terminal according to a second embodiment of the present invention.

It is noted that the technical terms used in the present invention are used only to describe specific embodiments and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be construed in a sense generally understood by a person having ordinary skill in the art to which the present invention belongs, unless otherwise defined in the present invention, Should not be construed to mean, or be interpreted in an excessively reduced sense. In addition, when a technical term used in the present invention is an erroneous technical term that does not accurately express the concept of the present invention, it should be understood that technical terms that can be understood by a person skilled in the art can be properly understood. In addition, the general terms used in the present invention should be interpreted according to a predefined or prior context, and should not be construed as being excessively reduced.

Furthermore, the singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. The term "comprising" or "comprising" or the like in the present invention should not be construed as necessarily including the various elements or steps described in the invention, Or may further include additional components or steps.

Furthermore, terms including ordinals such as first, second, etc. used in the present invention can be used to describe elements, but the elements should not be limited by terms. Terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or similar elements throughout the several views, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention, and should not be construed as limiting the scope of the present invention with reference to the accompanying drawings.

1 is a block diagram showing a configuration of a guide device 10 to which a terminal 100 according to an embodiment of the present invention is applied.

As shown in Fig. 1, the guide device 10 is composed of a terminal 100 and a server 200. Fig. Not all of the components of the guide device 10 shown in Fig. 1 are essential components, and the guide device 10 may be implemented by more components than the components shown in Fig. 1, The guiding device 10 may be realized.

The terminal 100 may be a smart phone, a portable terminal, a mobile terminal, a personal digital assistant (PDA), a portable multimedia player (PMP) terminal, a telematics terminal, A portable terminal, a navigation terminal, a personal computer, a notebook computer, a slate PC, a tablet PC, an ultrabook, a wearable device (for example, a watch type A smart TV, a digital broadcasting terminal, an AVN (Audio Video Navigation) terminal, a digital broadcasting terminal, a digital video broadcasting terminal, An audio / video (A / V) system, a flexible terminal, and the like.

2, the terminal 100 includes a communication unit 110, a camera 120, a storage unit 130, a display unit 140, an audio output unit 150, and a controller 160. Not all the components of the terminal 100 shown in Fig. 2 are essential components, and the terminal 100 may be implemented by more components than the components shown in Fig. 2, The terminal 100 may be implemented.

The communication unit 110 communicates with at least one terminal of an external arbitrary component through a wired / wireless communication network. At this time, the external arbitrary terminal may include the server 200 and the like. Here, the wireless Internet technology includes a wireless LAN (WLAN), a digital living network alliance (DLNA), a wireless broadband (Wibro), a world interoperability for a microwave (WiMAX), a high speed downlink packet access ), HSUPA (High Speed Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS) And the communication unit 110 transmits and receives data according to at least one wireless Internet technology in a range including Internet technologies not listed above. In addition, the near field communication technology includes Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, Near Field Communication (NFC) , Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, and Wi-Fi Direct. The wired communication technology may include a power line communication (PLC), a USB communication, an Ethernet, a serial communication, an optical / coaxial cable, and the like.

In addition, the communication unit 110 can transmit information with an arbitrary terminal through a universal serial bus (USB).

In addition, the communication unit 110 may be a mobile communication system such as a mobile communication system, a mobile communication system, a mobile communication system, a mobile communication system, a mobile communication system, (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (E.g., Long Term Evolution-Advanced), and the like, on the mobile communication network.

The communication unit 110 transmits the augmented reality image, additional information, and the like generated by the terminal 100 to the server 200 under the control of the controller 160. [

The communication unit 110 may control the current position of the terminal 100 transmitted from the server 200 or the keyword transmitted to the terminal 200 by the controller 200 under the control of the controller 160 Or a search word), additional information, and the like. Here, the additional information includes real-time additional information (or detailed information), related videos, coupons (or coupon information), and the like. At this time, the related video (or the promotional video) may be a video related to a specific product and / or service, a video related to the use of the product (e.g., a video about how to use cosmetics, A moving picture about how to use a tool, a moving picture about a cooking method, etc.), and may include characters and / or voice messages.

The camera 120 processes image frames such as still images or moving images obtained by an image sensor (a camera module or a camera) in a video communication mode, a shooting mode, a video conference mode, and the like. That is, the image data obtained by the image sensor according to the CODEC is encoded / decoded according to each standard. The processed image frame may be displayed on the display unit 140 under the control of the controller 160. [ For example, the camera 120 photographs an object (or a subject) (a user image or a video conference), and outputs a video signal corresponding to the photographed image (object image).

In addition, the camera 120 may store images (or two-dimensional images) including one or more stores on the off-line, merchandise (or display merchandise), POPs, flyers, banner advertisements, / Panoramic image). Here, the store includes a department store, a shopping mall, an outlet, a restaurant, a hospital, a building (or a building), and the like. At this time, the camera 120 may be a 360-degree camera for photographing a two-dimensional panoramic image.

As shown in FIG. 3, the display products indicate non-sale items for exhibition displayed for the sale of goods to the customer (for example, clothes put on mannequins, bags that are displayed on display shelves, etc.) ).

Also, as shown in FIG. 4, the POP (Point of Purchase) represents an advertisement providing information on goods and / or services on the spot to the customer who visited the store.

Also, as shown in FIG. 5, the leaflet (or the advertisement of the leaflet / flyer / leaflet) represents a form of a simple printed matter distributed for advertising or publicity purpose.

Also, as shown in FIG. 6, the banner advertisement is an advertisement provided at a specific location in a store to provide information on a specific product and / or service.

Also, as shown in FIG. 7, the poster billing refers to a paper made and adhered to an advertisement, and it is produced in consideration of characters, figures, and sentences so as to attract attention because it appeals to the time.

As shown in FIG. 8, the signboard indicates a plate indicating a company name, a store name, a product name, a service name, and the like with respect to a store.

The image frame processed by the camera 120 may be stored in the storage unit 130 or transmitted to an external terminal (for example, another terminal, the server 200, etc.) through the communication unit 110 have.

The storage unit 130 stores various user interfaces (UI), a graphical user interface (GUI), and the like.

In addition, the storage unit 130 stores data and programs necessary for the terminal 100 to operate.

That is, the storage unit 130 may store a plurality of application programs or applications that are driven by the terminal 100, data for operation of the terminal 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. At least some of these application programs may exist on the terminal 100 from the time of departure for the basic functions (e.g., call incoming, outgoing, message reception, origination) of the terminal 100. The application program may be stored in the storage unit 130 and may be installed in the terminal 100 and may be operated by the controller 160 to perform operations (or functions) of the terminal 100.

The storage unit 130 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD A random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory And a PROM (Programmable Read-Only Memory). Also, the terminal 100 may operate a web storage that performs a storage function of the storage unit 130 on the Internet, or may operate in association with the web storage.

The storage unit 130 may store the current position of the terminal 100 received through the communication unit 110 or the augmentation corresponding to the keyword transmitted to the terminal 200 to the terminal 200 A real image, and additional information.

The display unit 140 can display various contents such as various menu screens using the user interface and / or graphical user interface stored in the storage unit 120 under the control of the controller 160. [ Here, the content displayed on the display unit 140 includes various text or image data (including various information data) and a menu screen including data such as an icon, a list menu, and a combo box. Also, the display unit 140 may be a touch screen.

The display unit 140 may be a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) And may include at least one of a flexible display, a 3D display, an e-ink display, and an LED (Light Emitting Diode).

In addition, the display unit 140 displays an image photographed through the camera 120 under the control of the controller 160. [

In addition, the display unit 140 displays the augmented reality image generated by the control unit 160.

The display unit 140 may display the current position of the terminal 100 received through the communication unit 110 or the keyword transmitted to the terminal 200 by the controller 200 under the control of the controller 160 (Or a search word), additional information, and the like.

The audio output unit 150 outputs audio information included in the signal processed by the controller 160. [ Here, the audio output unit 150 may include a receiver, a speaker, a buzzer, and the like.

Also, the voice output unit 150 outputs the guidance voice generated by the control unit 160.

The voice output unit 150 may output the current position of the terminal 100 received through the communication unit 110 or the keyword transmitted to the terminal 200 by the controller 160 (Or a search word), audio information corresponding to additional information, and the like.

The control unit 160 executes the overall control function of the terminal 100. [

The control unit 160 executes the overall control functions of the terminal 100 using the programs and data stored in the storage unit 130. [ The control unit 160 may include a RAM, a ROM, a CPU, a GPU, and a bus, and the RAM, the ROM, the CPU, and the GPU may be connected to each other via a bus. The CPU accesses the storage unit 130 and performs booting using the O / S stored in the storage unit 130. The CPU 130 performs various operations using various programs, contents, and data stored in the storage unit 130 Can be performed.

In addition, the control unit 160 performs a membership registration procedure with the server 200 for the user of the terminal 100 by interlocking with the server 200.

In addition, when performing the membership registration procedure, the controller 160 must complete the authentication function through the authentication means (for example, a mobile phone, a credit card, an i-pin, Can normally be completed.

In addition, the control unit 160 provides the server 200 with user information such as date of birth, male / female information, telephone number, e-mail, password, name and address through the communication unit 110 do.

In addition, when an application installed in advance in the terminal 100 is executed, the controller 160 displays an application execution result screen on the display unit 140. [ Here, the application execution result screen may include a member information item (or menu / button) for inputting or transmitting member information, an image information item for providing an image or a 3D panorama image, a setting item for setting a function of the application, And a My Page item for managing purchase history information on a product or service purchased through the Internet 100. At this time, the terminal 100 communicates with the server 200 related to the corresponding application in the background state.

In addition, when a member information item is selected from the application execution result screen displayed on the display unit 140, the control unit 160 may display the member information for receiving (or receiving) the member information corresponding to the selected member information item And displays the screen on the display unit 140.

Also, the controller 160 receives member information according to user input (or user touch). Here, the member information includes a name, an address, a language used, a destination information, a shopping object type (for example, an avatar type, an emoticon type, etc.).

In addition, the controller 160 transmits the received membership information, identification information of the terminal 100, and the like to the server 200 through the communication unit 110. The identification information of the purchasing terminal 100 includes an MDN (Mobile Directory Number), a mobile IP, a mobile MAC, a SIM (subscriber identity module) card unique information, a serial number, and the like.

In addition, the control unit 160 recognizes one or more objects in the photographed image. Here, the object includes at least one store in the offline state, a product in the store, a POP, a flyer, a banner advertisement, a bulletin board advertisement, a poster advertisement, a signboard, a price tag and the like.

In addition, the control unit 160 inserts (or adds / adds) additional objects (for example, three-dimensional objects) related to the recognized object in the captured image to generate an augmented reality image. Here, the additional object may be a three-dimensional object, and may be an object for providing real-time additional information (or detailed information) related to the recognized object, a related moving image, a coupon, and the like.

That is, the control unit 160 identifies a three-dimensional additional object related to the recognized object.

In addition, the controller 160 inserts a three-dimensional additional object related to the recognized object in the two-dimensional captured image to generate an augmented reality image. Here, the augmented reality image may be composed of the captured image in a two-dimensional form and additional objects related to the recognized object in a three-dimensional form.

At this time, the light (or the specific color) of the recognized object and / or the additional object inserted in the image is changed to a maximum state (or a specific color) for the corresponding object and / , The effect of repeatedly changing from the extinction state to the maximum state, the flickering effect of the light, the neon sign effect, and the effect of changing the colors of the light to different colors at predetermined cycles may be applied.

In this way, the controller 160 generates interest and / or interest of the user in the two-dimensional image captured through the camera 120, and generates three-dimensional (Or adding) additional objects in the form of a new augmented reality image.

In addition, when generating the augmented reality image, various effects can be applied to the object, the additional object, and the like to facilitate identification in the augmented reality image.

Also, the controller 160 displays a specific augmented reality image on the display unit 140. [ The augmented reality image displayed on the display unit 140 corresponds to a keyword (or a search word) requested by the user of the terminal 100 and may be an augmented reality image previously stored in the storage unit 130, And may be an augmented reality image transmitted from the server 200 and received through the communication unit 110. [

When an additional object of a specific three-dimensional shape is selected in the augmented reality image displayed on the display unit 140, the control unit 160 controls the display unit 140 to display the selected three- And displays the additional information on the display unit 140. In this case, the additional information (or additional information related to the additional object in the three-dimensional form) may be information transmitted in real time from the server 200 through the communication unit 110.

That is, when a specific three-dimensional additional object in the augmented reality image displayed on the display unit 140 is selected, the control unit 160 may include real-time supplementary information related to the selected additional object, Identify additional information about the additional object.

In addition, the control unit 160 may include the identified additional information (for example, real-time additional information related to the additional object, a related video, a coupon, etc.) in a specific region (or a pop-up form) in the augmented reality image of the display unit 140 ).

At this time, the controller 160 may display the identified additional information in another specific area of the augmented reality image of the display unit 140 in a balloon shape or the like.

In addition, the controller 160 displays additional information related to the selected additional object (or an object corresponding to the additional object) adjacent to the selected additional object. At this time, the selected additional object (for example, additional object related to the product) displayed on the display unit 140, the object corresponding to the additional object (for example, a product corresponding to the additional object) The additional information (for example, the real-time additional information related to the additional object, the related moving image, the coupon, etc.) may be stored at a predetermined period so as to facilitate identification of the object, the additional object corresponding to the object, (Or a specific color) gradually extinguishes from the maximum state (or the lighted state) to the maximum state (or the lighted state) gradually from the lighted state to the object and / or the additional object, , A neon sign effect, and an effect that the colors of light are changed into different colors at predetermined cycles. Also, this effect may be maintained for a predetermined time.

In addition, in addition to selection (or touch / click) of additional three-dimensional objects in the augmented reality image displayed on the display unit 140, The additional information related to the additional object including the real-time additional information, the related moving image, the coupon, etc., corresponding to the additional object may be displayed on the display unit 140. [

At this time, the preset event is a state in which a touch gesture by the user of the terminal 100 for the additional object is sensed when the three-dimensional additional object is selected (or touched / clicked) And the like. The touch gesture by the user of the terminal 100 includes a tap, a touch & hold, a double tap, a drag, a flick, a drag and drop, a pinch, and a swipe.

Quot; tap "means an operation in which a user touches a screen (including an object, a place name, additional information, and the like) using a finger or a touch tool .

"Touch & hold" means that a user touches a screen (e.g., an object, a place name, additional information, etc.) using a finger or a touch tool 2 seconds) or more. That is, the time difference between the touch-in point and the touch-out point is equal to or greater than a threshold time (for example, 2 seconds). In order to allow the user to recognize whether the touch input is a tap or a touch & hold, the feedback signal may be provided visually, audibly or tactually when the touch input is maintained for a predetermined time or more. The threshold time may vary depending on the implementation.

"Double tap" represents an operation in which a user touches a screen (including an object, a place name, additional information, and the like) twice with a finger or a stylus.

A "drag" is a method in which a user touches a finger or a touch tool on a screen (e.g., an object, a place name, a side information, etc.) and then moves the finger or the touch tool to another position Operation. The object is moved due to the drag operation or a panning operation to be described later is performed.

A "flick" means that a user touches a finger or a touch tool to a screen (e.g., an object, a place name, additional information, etc.) ). ≪ / RTI > It is possible to distinguish the drag (or panning) from the flick based on whether the moving speed of the finger or the touch tool is equal to or greater than a critical speed (for example, 100 pixel / s).

"Drag & drop" means an operation in which a user drags an object (including an object, a place name, additional information, and the like) to a predetermined position using a finger or a touch tool and then releases the object.

A "pinch" indicates an operation in which a user moves two fingers in different directions while touching them on a screen (e.g., including an object, a name, and additional information).

A "swipe" is an operation of moving a certain distance in a horizontal or vertical direction while touching an object (for example, an object, a place name, additional information, etc.) on the screen with a finger or a touch tool. Motion in the diagonal direction may not be recognized as a swipe event.

In addition, when the function corresponding to the event is terminated, the controller 160 returns to the image before the event detection (or the augmented reality image).

That is, when the 3D object is selected, the control unit 160 displays the image (or the augmented reality image) before the event is displayed on the display unit 140.

If the predetermined time has elapsed since the touch gesture detection by the user of the terminal 100, the control unit 160 displays the image (or the augmented reality image) before the event is displayed on the display unit 140 do.

The server 200 communicates with the terminal 100 and the like.

In addition, the server 200 performs a membership registration procedure for a user of the terminal 100 by interworking with the terminal 100.

Also, the server 200 may perform an augmented reality image generation function performed in the AT 100. [

The augmented reality image generated according to the embodiment of the present invention includes positional information for an object, an additional object, a point (or a coordinate corresponding to the corresponding object and / or the additional object) included in the augmented reality image Interworking / matching).

That is, in order to search (or check) the augmented reality image corresponding to an arbitrary terminal (for example, including the position information of the terminal 100) in the future as well as the augmented reality image, Objects, additional objects, points, and the like.

Accordingly, when requesting an augmented reality image corresponding to the position information from the arbitrary terminal, the server 200 generates a plurality of augmented reality image information including positional information of one or more previously stored objects, additional objects, It is possible to confirm the augmented reality image information corresponding to the specific location information requested from the arbitrary terminal among the augmented reality image information and provide the augmented reality image information to the corresponding terminal.

For example, the server 200 may acquire, from the augmented reality image information associated with the location information of a plurality of stores and / or products, an object corresponding to the location information of the arbitrary terminal, which corresponds to 123-45 Gangnam- It is possible to confirm the augmented reality image information including the related additional object.

If there is no object corresponding to the location information of the arbitrary terminal and the additional object corresponding to 123-45 Gangnam-gu, Gangnam-gu, Seoul, from the augmented reality image information interlocked with the location information of the plurality of stores and / The server 200 may confirm the augmented reality image information including the object and / or the commodities similar to the position information of the arbitrary terminal (for example, 123-44 or 123-46 in Gangnam-gu, Gangnam-gu, Seoul) .

When the plurality of augmented reality image information is matched with the GPS information configured at predetermined intervals, the server 200 generates the arbitrary augmented reality image information based on the position information of the arbitrary terminal and the GPS information configured at the predetermined interval (Or similar / related) one of the augmented reality image information corresponding to the position information of the terminal.

In addition, the method of confirming the augmented reality image information corresponding to the specific position information among the plurality of augmented reality image information stored in advance in the server 200 may apply the techniques utilized in the map search service.

As described above, the augmented reality image information generated by the present invention is associated with the position information, and for a specific position information transmitted from an arbitrary terminal, among a plurality of augmented reality image information stored in advance in the server 200, It is possible to confirm the augmented reality image information corresponding to the specific location information and to provide the identified augmented reality image information to the corresponding terminal that has transmitted the specific location information.

Also, the server 200 receives a keyword (or a search word) transmitted from the terminal 100, identification information of the terminal 100, and the like.

Also, the server 200 identifies an augmented reality image corresponding to the keyword from a plurality of augmented reality images.

In addition, the server 200 transmits the augmented reality image corresponding to the identified keyword to the terminal 100.

When the additional object included in the augmented reality image provided to the terminal 100 is selected, the server 200 determines whether the additional object included in the augmented reality image is included in the augmented reality image, And provides the terminal 100 with additional information. At this time, the additional information (or additional information related to the additional object in the three-dimensional form) includes real-time additional information related to the additional object, related moving image, coupon (or coupon information), and the like.

When the additional object (or the additional object to which the predetermined effect is applied) is selected in the augmented reality image displayed on the current screen of the terminal 100, the server 200 transmits a request to the terminal 100 Related information related to the additional object selected by the terminal 100, related video, coupon (or coupon information), and the like to the terminal 100 in real time.

In this manner, a 2D panoramic image can be captured, an object can be recognized in the panoramic image, and an additional 3D object in association with the recognized object can be inserted to generate an augmented reality image.

Also, in the case where the 3D object added in the augmented reality image is selected, real-time additional information related to the additional object can be displayed.

Hereinafter, a method of controlling a terminal according to the present invention will be described in detail with reference to FIGS. 1 to 21. FIG.

9 is a flowchart illustrating a method of controlling a terminal according to a first embodiment of the present invention.

First, the camera 120 captures an image (or a two-dimensional image / panoramic image) including one or more stores on the offline line, products in the store, POP, flyer, banner advertisement, bulletin board advertisement, poster advertisement, signboard, do. Here, the store includes a department store, a shopping mall, an outlet, a restaurant, a hospital, a building (or a building), and the like. At this time, the camera 120 may be a 360-degree camera for photographing a two-dimensional panoramic image.

The display unit 140 displays an image photographed through the camera 120 under the control of the control unit 160. [

For example, the camera 120 photographs an eleventh image including a hat and glasses as an exhibition product in a large shopping mall. Also, as shown in FIG. 10, the display unit 140 displays the photographed eleventh image 1000.

In another example, the camera 120 photographs a twelfth image including a plurality of boutiques on the off-line. Also, as shown in FIG. 11, the display unit 140 displays the photographed 12th image 1100 (S910).

Then, the control unit 160 recognizes the object in the photographed image. Here, the object includes at least one store in the offline state, a product in the store, a POP, a flyer, a banner advertisement, a bulletin board advertisement, a poster advertisement, a signboard, a price tag and the like.

For example, the control unit 160 recognizes the hat and glasses included in the photographed eleventh image, respectively.

In other words, the control unit 160 recognizes a plurality of stores (for example, ABC hotel, DEF hotel, etc.) included in the photographed 12th image (S920).

Then, the control unit 160 inserts (or adds) additional objects (for example, three-dimensional objects) related to the recognized object in the captured image to generate an augmented reality image. Here, the additional object may be a three-dimensional object, and may be an object for providing real-time additional information (or detailed information) related to the recognized object, a related moving image, a coupon (or coupon information)

That is, the control unit 160 identifies a three-dimensional additional object related to the recognized object.

In addition, the control unit 160 generates an augmented reality image by inserting a three-dimensional additional object related to the recognized recognized object adjacent to the recognized object in the captured two-dimensional image do. Here, the augmented reality image may be composed of the captured image in a two-dimensional form and additional objects related to the recognized object in a three-dimensional form.

At this time, the light (or the specific color) of the recognized object and / or the additional object inserted in the image is changed to a maximum state (or a specific color) for the corresponding object and / , The effect of repeatedly changing from the extinction state to the maximum state, the flickering effect of the light, the neon sign effect, and the effect of changing the colors of the light to different colors at predetermined cycles may be applied.

For example, the control unit 160 may display the real time additional information (or product detailed information) related to the hat 1210 included in the photographed eleventh image, the related video, the coupon (or coupon information) A twenty-second additional object 1240 for identifying the additional object 1220 and displaying real-time additional information related to the glasses 1230 included in the photographed eleventh image, related videos, coupons (or coupon information) Check. 12, the controller 160 inserts the identified twenty-first additional object 1220 and the twenty-second additional object 1240 into the eleventh image 1000, (1250). In this case, the first augmented reality image may be formed of an eleventh image of a two-dimensional shape and a twenty-first additional object and a twenty-second additional object of a three-dimensional shape. In addition, the hat in the first augmented reality image is a state in which the light gradually extinguishes at a preset period and the effect of repeatedly changing the light from the extinction state to the maximum state is applied (in FIG. 12, And displayed in a state in which the pattern is applied to the hat in the augmented reality image).

The control unit 160 may receive the real-time reservation status information (or the real-time supplementary information) related to the ABC hotel 1310 included in the photographed twelfth image, the related video (or promotional video), the coupon The real time reservation status information (or real time additional information) related to the DEF hotel 1330 included in the photographed 12th video, the related video (or the publicity video ), And so on. ≪ / RTI > 13, the controller 160 inserts the identified 31st additional object 1320 and 32nd additional object 1340 into the 12th video 1100, (1350). In this case, the second augmented reality image may be formed of a twelfth image of two-dimensional shape and a thirty-first additional object and a thirty-second additional object of three-dimensional shape. Also, in the second augmented reality image, the ABC hotel and the DEF hotel are gradually extinguished at a preset period of time at a maximum state and are repeatedly changed to a maximum state gradually from the unlit state to the maximum state (A pattern is applied to the ABC hotel and the DEF hotel in the second augmented reality image) (S930).

Fig. 14 6 is a flowchart illustrating a method of controlling a terminal according to a second embodiment of the present invention.

First, the display unit 140 displays a specific augmented reality image under the control of the control unit 160. [ At this time, the augmented reality image corresponds to a keyword (or a search word) requested by a user of the AT 100, and may be stored in the AT 100 in advance, or may be stored in a state It is possible.

15, the display unit 140 displays the first augmented reality image 1250 stored in advance in the storage unit 130 under the control of the control unit 160. As shown in FIG. Here, the first augmented reality image includes a 21-th addition in a three-dimensional form for displaying a two-dimensional hat and glasses image, real-time additional information related to the hat, a related animation, a coupon (or coupon information) And a twenty-second additional object in a three-dimensional form for indicating real-time additional information related to the eyeglasses, a related moving image, a coupon (or coupon information), and the like. In addition, the hat in the first augmented reality image is a state in which the light gradually extinguishes at a preset period and the effect of repeatedly changing the light from the extinction state to the maximum state is applied (in FIG. 12, And displayed in a state in which the pattern is applied to the hat in the augmented reality image).

In another example, under control of the controller 160, the communication unit 110 receives a second augmented reality image transmitted in real time from the server 200 in association with a specific keyword (for example, Yeoksam station distance) . 16, the display unit 140 displays the second augmented reality image 1350 received through the communication unit 110 under the control of the controller 160. [ Here, the second augmented reality image includes a video including a plurality of stores in a two-dimensional offline form (for example, an ABC hotel, a DEF hotel, etc.), a reservation status information related to the ABC hotel, (Or a DEF hotel publicity image) related to the DEF hotel, and a 31st additional object in the form of a three-dimensional figure for displaying the DEF hotel (or the ABC hotel promotion image), coupon (or coupon information) , A coupon (or coupon information), and the like. Also, in the second augmented reality image, the ABC hotel and the DEF hotel are gradually extinguished at a preset period of time at a maximum state and are repeatedly changed to a maximum state gradually from the unlit state to the maximum state (A pattern is applied to the ABC hotel and the DEF hotel in the second augmented reality image) (S1410).

If the additional three-dimensional object in the augmented reality image displayed on the display unit 140 is selected, the controller 160 adds a specific three-dimensional object to the specific area in the augmented reality image And displays the additional information on the display unit 140. In this case, the additional information (or additional information related to the additional object in the three-dimensional form) may be information transmitted in real time from the server 200 through the communication unit 110.

That is, when a specific three-dimensional additional object in the augmented reality image displayed on the display unit 140 is selected, the controller 160 adds real-time supplementary information, related videos, coupons (or coupon information ), And so on.

In addition, the control unit 160 may add the identified additional information (e.g., real-time additional information related to the additional object, related video, coupon information, etc.) to a specific region (or a pop- ).

For example, when the real-time additional information object (or real-time additional information) 1221 included in the twenty-first additional object related to the hat in the first augmented reality image shown in FIG. 15 is selected, And transmits a first request signal for real time additional information related to the cap to the server 200 through the communication unit 110. [ Then, the controller 160 transmits real-time additional information (for example, product detailed information on a hat, etc.) related to the hat, which is transmitted from the server 200, to the communication unit 110 in response to the first request signal. Lt; / RTI > Then, as shown in FIG. 17, the controller 160 transmits real-time additional information (for example, product detailed information on a hat and the like) 1710 related to the received hat to the first augmented reality image Is displayed in the first area.

15, when the related moving image object 1222 included in the twenty-first additional object related to the hat in the first augmented reality image shown in FIG. 15 is selected, the control unit 160 controls the related moving image related to the hat And transmits the second request signal to the server 200 via the communication unit 110. Thereafter, the controller 160 receives a related video (e.g., a video related to the cap) related to the cap transmitted from the server 200 via the communication unit 110 in response to the second request signal . 18, the controller 160 displays a related video (e.g., a video related to the hat) 1810 related to the received hat in a second area of the first augmented reality image do.

As another example, when the coupon 1223 included in the twenty-first additional object related to the hat in the first augmented reality image shown in FIG. 15 is selected, as shown in FIG. 19, the control unit 160 And coupon information 1910 corresponding to the coupon associated with the hat is displayed in the third area in the first augmented reality image.

In another example, when the reservation status information 1321 included in the 31st additional object related to the ABC hotel in the second augmented reality image shown in FIG. 16 is selected, the control unit 160 determines And transmits a third request signal for the real-time reservation status information to the server 200 through the communication unit 110. [ In response to the third request signal, the controller 160 transmits real-time reservation status information related to the ABC hotel transmitted from the server 200 (for example, the total number of rooms: 300, which is reservation status information of the ABC hotel) And the number of rooms available for reservation: 150) through the communication unit 110. Then, as shown in FIG. 20, the controller 160 stores real-time reservation status information related to the received ABC hotel (for example, 300 total rooms, reservation status information of the ABC hotel, : 150) 2010 are displayed in the fourth area in the second augmented reality image.

16, when the related moving image included in the 31st additional object related to the ABC hotel in the second augmented reality image shown in FIG. 16 is selected, the control unit 160 controls the moving image related to the ABC hotel And transmits the fourth request signal to the server 200 through the communication unit 110. [ Thereafter, the controller 160 transmits a publicity image (or an ABC hotel publicity image) of a real-time ABC hotel related to the ABC hotel transmitted from the server 200 in response to the fourth request signal to the communication unit 110 Lt; / RTI > 21, the control unit 160 transmits a publicity image (or an ABC hotel publicity image) 2110 for a real-time ABC hotel related to the received ABC hotel, 5 area (S1420).

As described above, in the embodiment of the present invention, the panoramic image of the 2D form is photographed, the object is recognized in the photographed panoramic image, and the additional object in 3D form related to the recognized object is inserted, So that it is possible to easily provide additional information related to the specific object.

In addition, in the embodiment of the present invention, as described above, when a 3D object in the 3D form inserted in the augmented reality image is selected, real-time additional information related to the additional object is displayed to provide a user interface for user convenience And it is possible to improve the user's convenience in providing the additional information related to the object of interest to the user on the current screen.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or essential characteristics thereof. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

The present invention relates to a method of generating an augmented reality image by capturing a 2D panoramic image, recognizing an object in the panoramic image, inserting an additional object in 3D form related to the recognized object, When the inserted 3D object is selected, real-time additional information related to the additional object is displayed, thereby providing a user interface for user's convenience and providing additional information related to the object of interest on the current screen And can be widely used in a terminal field, a panoramic field field, a shopping information field, a marketing field, a server field, and the like.

10: guide device 100: terminal
200: server 110:
120: camera 130: storage unit
140: display unit 150: audio output unit
160:

Claims (1)

A method for controlling a terminal,
A step of photographing a two-dimensional image by a camera;
Displaying the photographed image by a display unit;
Generating augmented reality image by inserting a three-dimensional additional object into the photographed two-dimensional image by the control unit; And
And displaying the generated augmented reality image by the display unit,
The additional object may further comprise:
Wherein the object is an object for providing real-time additional information related to an object included in the photographed image, a related moving image, and coupon information.

Images