KR20140019966A - Mobile terminal and controlling method thereof, and recording medium thereof - Google Patents

Abstract

The present invention provides a mobile terminal including a memory; a wireless communication unit; a camera for generating a preview image of surroundings; a touch screen for displaying the preview image and a virtual object of a first wireless communication terminal in the preview image; and a control unit controlling the mobile terminal in order for data communication to be performed between a first wireless communication terminal which a first note indicates and the wireless communication unit, when the first note is inputted on the touch screen, or the first communication terminal and a second communication terminal. [Reference numerals] (S401) Activate a camera; (S402) Display a preview image of surroundings; (S403) Display mixed reality video; (S404) Entry a handwriting input mode; (S405) First handwriting input; (S406) User's handwriting is ended?; (S407) Control a first wireless communication terminal and a wireless communication unit or a second wireless communication terminal in order for data communication to be performed

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a mobile terminal and a control method thereof,

The present invention relates to a mobile terminal configured to perform data communication between a mobile communication terminal and a wireless communication terminal captured on a touch screen in an augmented reality environment, and a control method thereof.

A terminal can be divided into a mobile terminal (mobile / portable terminal) and a stationary terminal according to whether the terminal can be moved. The mobile terminal may be further classified into a handheld terminal and a vehicle mount terminal according to whether a user can directly carry it.

Such a terminal has various functions, for example, in the form of a multimedia device having multiple functions such as photographing and photographing of a moving picture, reproduction of a music or video file, reception of a game and broadcasting, etc. .

Recently, an augmented reality service for controlling a display of a mixed reality image in which a preview image input through a camera of a mobile terminal and a virtual object having additional information of a reality object captured in the preview image is displayed is widely implemented.

However, the existing augmented reality service is merely for displaying additional information of the reality object captured in the preview image. Accordingly, there is an attempt to extend the function of the mobile terminal by using additional information of the reality object, away from the existing augmented reality service.

In this trend, the present invention intends to propose an interworking method between a real object captured in a preview image and a mobile terminal while using an augmented reality service.

The present invention is to provide a mobile terminal with improved user convenience.

Specifically, the present invention is to provide a mobile terminal that can easily share data between the wireless communication terminal and the mobile terminal in the mixed reality image.

Furthermore, the present invention is to provide a mobile terminal capable of data sharing between the wireless communication terminal and the mobile terminal through a user's handwriting input.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, unless further departing from the spirit and scope of the invention as defined by the appended claims. It will be possible.

A mobile terminal according to an embodiment of the present invention, a memory; A wireless communication unit; A camera for generating a preview image of the surrounding environment; A touch screen for displaying a virtual object of the first wireless communication terminal in the preview image and the preview image; And when the first handwriting is input to the touch screen, data between the first wireless communication terminal indicated by the first handwriting and the second wireless communication terminal between the wireless communication unit or the first wireless communication terminal and the second wireless communication terminal. It may include a control unit for controlling the communication to be performed.

The technical solutions obtained by the present invention are not limited to the above-mentioned solutions, and other solutions not mentioned are clearly described to those skilled in the art from the following description. It can be understood.

The mobile terminal according to the present invention can be expected to improve the user's operation convenience.

Specifically, the present invention can provide a mobile terminal that can easily share data between the wireless communication terminal and the mobile terminal in the mixed reality image.

Furthermore, the present invention can provide a mobile terminal capable of data sharing between the wireless communication terminal and the mobile terminal through a user's handwriting input.

The effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description will be.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.
2 is a front perspective view of an example of a mobile terminal or a mobile terminal according to the present invention;
3 is a rear perspective view of the mobile terminal shown in FIG. 2;
4 is an operation flowchart of a mobile terminal according to an embodiment of the present invention;
5 is an exemplary view of displaying a mixed reality image in which a preview image and a virtual object are fused.
6 is a diagram illustrating an output of a touch screen for controlling data communication between a first wireless communication terminal and a second wireless communication terminal.
FIG. 7 is an exemplary diagram for controlling data communication between a first wireless communication terminal displayed in the mixed reality image and a second wireless communication terminal not displayed in the mixed reality image; FIG.
8 is an exemplary diagram for explaining that data communication is performed between a first wireless communication terminal and a wireless communication unit.
9 is an exemplary diagram for describing an operation of a mobile terminal when a reality object is a face of a specific counterpart.
10 is an exemplary diagram for explaining that a wireless communication unit is set to communicate with a plurality of wireless communication terminals.
11 is an exemplary diagram for explaining that additional information of a reality object is transmitted to a specific counterpart;
FIG. 12 is an exemplary diagram for describing an operation of a mobile terminal for controlling a purchase of a reality object using a virtual object in a reality mixed image; FIG.
FIG. 13 illustrates an example in which visual effects are applied to a reality object in a mixed reality image. FIG.

Hereinafter, a mobile terminal related to the present invention will be described in detail with reference to the drawings. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role.

The mobile terminal described in this specification may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), and navigation. However, it will be readily apparent to those skilled in the art that the configuration according to the embodiments described herein may also be applied to fixed terminals such as digital TVs, desktop computers, etc., except when applicable only to mobile terminals.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an audio / video input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, A controller 170, a controller 180, a power supply 190, and the like. The components shown in FIG. 1 are not essential, and a mobile terminal having more or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules for enabling wireless communication between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and the network in which the mobile terminal 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short range communication module 114, and a location information module 115 .

The broadcast receiving module 111 receives a broadcast signal and / or broadcast related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast reception modules 1100 may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The broadcast management server may refer to a server for generating and transmitting broadcast signals and / or broadcast related information, or a server for receiving broadcast signals and / or broadcast related information generated by the broadcast management server and transmitting the generated broadcast signals and / or broadcast related information. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

The broadcast-related information means information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112.

The broadcast-related information may exist in various forms. For example, an EPG (Electronic Program Guide) of DMB (Digital Multimedia Broadcasting) or an ESG (Electronic Service Guide) of Digital Video Broadcast-Handheld (DVB-H).

For example, the broadcast receiving module 111 may be a Digital Multimedia Broadcasting-Terrestrial (DMB-T), a Digital Multimedia Broadcasting-Satellite (DMB-S), a Media Forward Link Only (Mobile Broadcasting Business Management System), ISDB-T (Integrated Services Digital Broadcasting (ISDB-T)), Digital Multimedia Broadcasting (MBMS) Digital Broadcast-Terrestrial) or the like. Of course, the broadcast receiving module 111 may be adapted to other broadcasting systems as well as the digital broadcasting system described above.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 112 may be coupled to a base station, an external terminal, or a server on a mobile communication network, such as, but not limited to, Gobal System for Mobile communications (GSM), Code Division Multiple Access (CDMA), Wideband CDMA Transmits and receives wireless signals with one. The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be built in or externally attached to the mobile terminal 100. Wireless Internet technologies include WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), GSM, CDMA, WCDMA, LTE Evolution (but not limited to) may be used.

The wireless Internet module 113, which performs wireless Internet access through the mobile communication network, is connected to the mobile communication module 110 through the mobile communication network, for example, from the viewpoint that the wireless Internet access by Wibro, HSDPA, GSM, CDMA, WCDMA, LTE, (112). ≪ / RTI >

The short-range communication module 114 refers to a module for short-range communication. Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra Wideband (UWB), ZigBee, and the like can be used as a short range communication technology.

The position information module 115 is a module for obtaining the position of the mobile terminal, and a representative example thereof is a Global Position System (GPS) module. According to the current technology, the GPS module 115 calculates distance information and accurate time information from three or more satellites, and then applies trigonometry to the calculated information to obtain a three-dimensional string of latitude, longitude, The location information can be accurately calculated. At present, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another satellite is widely used. In addition, the GPS module 115 can calculate speed information by continuously calculating the current position in real time.

Referring to FIG. 1, an A / V (Audio / Video) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame can be displayed on the display unit 151. [

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ Two or more cameras 121 may be provided depending on the use environment.

The microphone 122 receives an external sound signal through a microphone in a communication mode, a recording mode, a voice recognition mode, or the like, and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 112 when the voice data is in the call mode, and output. Various noise reduction algorithms may be implemented in the microphone 122 to remove noise generated in receiving an external sound signal.

The user input unit 130 generates input data for a user to control the operation of the terminal. The user input unit 130 may include a key pad, a dome switch, a touch pad (static / static), a jog wheel, a jog switch, and the like.

The sensing unit 140 senses the current state of the mobile terminal 100 such as the open / close state of the mobile terminal 100, the position of the mobile terminal 100, the presence or absence of user contact, the orientation of the mobile terminal, And generates a sensing signal for controlling the operation of the mobile terminal 100. For example, when the mobile terminal 100 is in the form of a slide phone, it may sense whether the slide phone is opened or closed. In addition, whether the power supply unit 190 is supplied with power, whether the interface unit 170 is coupled to the external device may be sensed. Meanwhile, the sensing unit 140 may include a proximity sensor 141.

The output unit 150 is for generating output related to the visual, auditory or tactile sense and includes a display unit 151, an audio output module 152, an alarm unit 153, and a haptic module 154 .

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, when the mobile terminal is in the call mode, a UI (User Interface) or a GUI (Graphic User Interface) associated with a call is displayed. When the mobile terminal 100 is in the video communication mode or the photographing mode, the photographed and / or received video or UI and GUI are displayed.

The display unit 151 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display display, and a 3D display.

Some of these displays may be transparent or light transmissive so that they can be seen through. This can be referred to as a transparent display, and a typical example of the transparent display is TOLED (Transparent OLED) and the like. The rear structure of the display unit 151 may also be of a light transmission type. With this structure, the user can see an object located behind the terminal body through the area occupied by the display unit 151 of the terminal body.

There may be two or more display units 151 according to the embodiment of the mobile terminal 100. For example, in the mobile terminal 100, a plurality of display portions may be spaced apart from one another, or may be disposed integrally with one another, and may be disposed on different surfaces, respectively.

(Hereinafter, referred to as a 'touch screen') in which a display unit 151 and a sensor for sensing a touch operation (hereinafter, referred to as 'touch sensor') form a mutual layer structure, It can also be used as an input device. The touch sensor may have the form of, for example, a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in a pressure applied to a specific portion of the display unit 151 or a capacitance generated in a specific portion of the display unit 151 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the pressure at the time of touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller (not shown). The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. As a result, the controller 180 can know which area of the display unit 151 is touched.

The proximity sensor 141 may be disposed in an inner region of the mobile terminal or in the vicinity of the touch screen, which is enclosed by the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or a nearby object without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor has a longer life span than the contact sensor and its utilization is also high.

Examples of the proximity sensor include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. And to detect the proximity of the pointer by the change of the electric field along the proximity of the pointer when the touch screen is electrostatic. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the act of recognizing that the pointer is positioned on the touch screen while the pointer is not in contact with the touch screen is referred to as "proximity touch" The act of actually contacting the pointer on the touch screen may be referred to as "contact touch. &Quot; The location where the pointer is proximately touched on the touch screen may refer to a position where the pointer corresponds vertically to the touch screen when the pointer is touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, and the like). Information corresponding to the detected proximity touch operation and the proximity touch pattern may be output on the touch screen.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output module 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The audio output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event of the mobile terminal 100. Examples of events that occur in the mobile terminal include call signal reception, message reception, key signal input, touch input, and the like. The alarm unit 153 may output a signal for notifying the occurrence of an event in a form other than the video signal or the audio signal, for example, vibration. In this case, the display unit 151 and the sound / voice output module 152 may be a kind of the alarm unit 153. The display unit 151 and the sound / . ≪ / RTI >

The haptic module 154 generates various tactile effects that the user can feel. Vibration is a representative example of the haptic effect generated by the haptic module 154. The intensity and pattern of vibration generated by the haptic module 154 can be controlled. For example, different vibrations may be synthesized and output or sequentially output.

In addition to the vibration, the haptic module 154 may include a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, contact with an electrode, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 154 can be implemented not only to transmit the tactile effect through the direct contact but also to allow the user to feel the tactile effect through the muscular sensation of the finger or arm. At least two haptic modules 154 may be provided according to the configuration of the mobile terminal 100.

The memory unit 160 may store a program for processing and controlling the control unit 180 and temporarily store the input / output data (e.g., telephone directory, message, audio, For example. The memory unit 160 may store the frequency of use of each of the data (for example, each telephone number, each message, and frequency of use for each multimedia).

In addition, the memory unit 160 may store data on vibration and sound of various patterns output when the touch is input on the touch screen.

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), a RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM A disk, and / or an optical disk. The mobile terminal 100 may operate in association with a web storage that performs a storage function of the memory 160 on the Internet.

The interface unit 170 serves as a path for communication with all external devices connected to the mobile terminal 100. The interface unit 170 receives data from an external device or supplies power to each component in the mobile terminal 100 or transmits data to the external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, A video input / output (I / O) port, an earphone port, and the like may be included in the interface unit 170.

The identification module is a chip for storing various information for authenticating the use right of the mobile terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM), a general user authentication module A Universal Subscriber Identity Module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the port.

When the mobile terminal 100 is connected to an external cradle, the interface unit may be a path through which power from the cradle is supplied to the mobile terminal 100, or various command signals input by the user to the cradle may be transmitted It can be a passage to be transmitted to the terminal. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the mobile terminal. For example, voice communication, data communication, video communication, and the like. The control unit 180 may include a multimedia module 181 for multimedia playback. The multimedia module 181 may be implemented in the control unit 180 or may be implemented separately from the control unit 180. [

The controller 180 may perform a pattern recognition process for recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively.

The power supply unit 190 receives an external power source and an internal power source under the control of the controller 180 to supply power for operation of each component.

The various embodiments described herein may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and the like. It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code is stored in the memory 160 and can be executed by the control unit 180. [

FIG. 2 is a perspective view of a mobile terminal or a portable terminal according to the present invention, as viewed from the front.

The disclosed mobile terminal 100 includes a bar-shaped terminal body. However, the present invention is not limited thereto, and can be applied to various structures such as a slide type, a folder type, a swing type, and a swivel type in which two or more bodies are relatively movably coupled.

The body includes a case (a casing, a housing, a cover, and the like) which forms an appearance. In this embodiment, the case may be divided into a front case 101 and a rear case 102. [ A variety of electronic components are embedded in the space formed between the front case 101 and the rear case 102. At least one intermediate case may be additionally disposed between the front case 101 and the rear case 102. [

The cases may be formed by injection molding a synthetic resin, or may be formed to have a metal material such as stainless steel (STS) or titanium (Ti) or the like.

The display unit 151, the sound output unit 152, the camera 121, the user input units 130/131 and 132, the microphone 122, the interface 170, and the like may be disposed in the front body 101 have.

The display unit 151 occupies most of the main surface of the front case 101. A sound output unit 151 and a camera 121 are disposed in an area adjacent to one end of both ends of the display unit 151 and a user input unit 131 and a microphone 122 are disposed in an area adjacent to the other end. The user input unit 132 and the interface 170 may be disposed on the side surfaces of the front case 101 and the rear case 102. [

The user input unit 130 is operated to receive a command for controlling the operation of the mobile terminal 100 and may include a plurality of operation units 131 and 132. The operation units 131 and 132 may be collectively referred to as a manipulating portion.

The contents inputted by the first or second operation unit 131 or 132 may be variously set. For example, the first operation unit 131 receives commands such as start, end, scroll, and the like, and the second operation unit 132 controls the size of the sound output from the sound output unit 152 or the size of the sound output from the display unit 151 The touch recognition mode can be activated or deactivated.

3 is a rear perspective view of the mobile terminal shown in FIG. 2.

Referring to FIG. 3, a camera 121 'may be further mounted on the rear surface of the terminal body, that is, the rear case 102. The camera 121 'may have a photographing direction substantially opposite to that of the camera 121 (see FIG. 2A), and may be a camera having the same or different pixels as the camera 121.

For example, the camera 121 may have a low pixel so that the face of the user can be photographed and transmitted to the other party in case of a video call or the like, and the camera 121 ' It is preferable to have a large number of pixels. The cameras 121 and 121 'may be installed in the terminal body so as to be rotatable or pop-upable.

A flash 123 and a mirror 124 may be additionally disposed adjacent to the camera 121 '. The flash 123 illuminates the subject when the subject is photographed by the camera 121 '. The mirror 124 allows the user to illuminate the user's own face or the like when the user intends to shoot himself / herself (self-photographing) using the camera 121 '.

An acoustic output 152 'may be additionally disposed on the rear surface of the terminal body. The sound output unit 152 'may implement the stereo function together with the sound output unit 152 (see FIG. 2A), and may be used for the implementation of the speakerphone mode during a call.

In addition to the antenna for communication, a broadcast signal receiving antenna 116 may be additionally disposed on the side of the terminal body. The antenna 116, which forms part of the broadcast receiving module 111 (see FIG. 1), can be installed to be able to be drawn out from the terminal body.

A power supply unit 190 for supplying power to the mobile terminal 100 is mounted on the terminal body. The power supply unit 190 may be built in the terminal body or may be detachable from the outside of the terminal body.

The rear case 102 may further include a touch pad 135 for sensing a touch. The touch pad 135 may be of a light transmission type for the display unit 151. [ In this case, if the display unit 151 is configured to output the time information on both sides (i.e., in the directions of both the front and back sides of the mobile terminal), the time information can be recognized through the touch pad 135 do. The information output on both sides may be all controlled by the touch pad 135. [

Meanwhile, the display for exclusive use of the touch pad 135 may be separately installed, so that the touch screen may be disposed in the rear case 102 as well.

The touch pad 135 operates in association with the display unit 151 of the front case 101. The touch pad 135 may be disposed parallel to the rear of the display unit 151. The touch pad 135 may have a size equal to or smaller than that of the display unit 151.

Hereinafter, embodiments related to a control method that can be implemented in the mobile terminal 100 configured as described above will be described with reference to the accompanying drawings.

For convenience of description, it is assumed that the mobile terminal 100 according to the present invention includes at least one of the components shown in FIG. 1. In detail, the mobile terminal 100 according to the present invention may include a wireless communication unit 110, a camera 121, a display unit 151, a memory 160, and a controller 180. The mobile terminal 100 according to the present invention can be implemented more easily in the case where the display unit 151 is the touch screen 151. Hence, it is assumed that the display unit 151 is the touch screen 151 in the following description.

First, the mobile terminal 100 according to the present invention will be described in detail through the operation flowchart of FIG. 4.

4 is an operation flowchart of a mobile terminal 100 according to an embodiment of the present invention. Referring to FIG. 4, first, the mobile terminal 100 may activate the camera 121 under the control of the controller 180 (S401). When the camera 121 is activated, the controller 180 can control a preview image of the surrounding environment captured by the camera 121 to be displayed on the touch screen 151 (S402). The preview image is an image that is continuously contained in the camera 121 after the camera 121 module is activated, and is different from the captured image photographed as the photographing command is applied. The preview image may be understood as representing the real world for applying Augmented Reality technology.

The controller 180 may apply the augmented reality technology to the preview image so as to display a mixed reality image in which a real object included in the preview image and a virtual object having additional information of the reality object are merged into one image (see FIG. S403).

For example, FIG. 5 is an exemplary diagram of displaying a mixed reality image in which a preview image and a virtual object are fused. As in the example shown in FIG. 5, the controller 180 has a reality object 501 (shown in FIG. 5 as a barcode of a gift card) in the preview image 510 and additional information associated with the reality object 501. The mixed reality image 530 obtained by fusing the virtual object 520 into one image may be displayed.

In displaying the virtual object 520 in the mixed reality image 530, the controller 180 may control to display the virtual object 520 in the form of a button associated with an access address to additional information. For example, in the example of FIG. 5, when the detail information button is pressed, the controller 180 may attempt to access a web page on which detailed information of an object in the preview image 510 is recorded.

The additional information about the reality object may be obtained from the augmented reality server, or may be obtained from stored data in the memory 160 of the mobile terminal.

The augmented reality server may receive the preview image or the location information of the mobile terminal 100 from the mobile terminal 100 and extract additional information corresponding to the reality object in the preview image. When the additional information extracted from the augmented reality server is received through the wireless communication unit 110, the controller 180 may display a mixed reality image simultaneously displaying a preview image and a virtual object having the additional information.

As another example, the controller 180 may obtain additional information of the real object from data stored in the memory 160. For example, when the reality object in the preview image is the face of the specific counterpart or the wireless communication terminal of the specific counterpart, the controller 180 may determine, from the phone book, the name of the specific counterpart, the name of the wireless communication terminal, the phone number, the email address, At least one of a social network service (SNS) identification and an instant message (IM) service ID may be extracted to control the wireless communication terminal of a specific counterpart to be a subject of data communication. If the phone book of the user of the mobile terminal 100 is also stored in the augmented reality server, the communication account of the reality object may be extracted from the augmented reality server.

For convenience of description, the reality object and the virtual object in the preview image will be collectively referred to as 'augmented reality object'. The augmented reality object to be used herein may mean any one of a reality object and a virtual object in the preview image, or may mean both a reality object and a virtual object in the preview image.

If the additional information about the reality object relates to the communication account of the reality object, the controller 180 may control the reality object to be a subject of data communication. In detail, when a user command is input while displaying the mixed reality image through the touch screen 151, the controller 180 may enter the handwriting input mode based on the user command (S404). After the first handwriting is input in the handwriting input mode (S405), if it is determined that the user's touch input is terminated (S406), the controller 180 determines that the first wireless communication terminal and the second wireless communication terminal designated by the first handwriting Alternatively, data communication may be performed between the first wireless communication terminal and the wireless communication unit 110 (S407).

Data communication between the first wireless communication terminal and the second wireless communication terminal will be described in more detail with reference to FIG. 6.

6 is a diagram illustrating an output of a touch screen 151 for controlling data communication between a first wireless communication terminal and a second wireless communication terminal. For convenience of explanation, it is assumed that the first wireless communication terminal 10 is a smartphone and the second wireless communication terminal 20 is a notebook.

The user selects the first wireless communication terminal 10 in the mixed reality image and selects the first wireless communication terminal 10 and the second wireless communication terminal 20, as in the example shown in FIG. 6A. The first handwriting 601 may be input in the form of a line segment having both an augmented reality object and an augmented reality object of the second wireless communication terminal 20. At this time, the first handwriting 601 is unidirectional, starting with (or ending with) the first wireless communication terminal 10 and ending with (or starting with) the second wireless communication terminal 20, as in the example shown in FIG. 6A. 6, the first handwriting 601 may be a virtual object ('smart phone' button) of the first wireless communication terminal 10 and a virtual object ('in' of the second wireless communication terminal 20). Laptop 'button).

When the first handwriting 601 input is completed, the controller 180 controls data communication to be performed between the first wireless communication terminal 10 and the second wireless communication terminal 20 indicated by the first writing 601 input. can do.

In controlling to perform data communication between the first wireless communication terminal 10 and the second wireless communication terminal 20, the controller 180 may determine the data transmission direction in a direction designated by the first writing 601. Can be controlled. For example, as in the example illustrated in FIG. 6A, when the first writing 601 is a unidirectional arrow, the one-sided arrow may be controlled to function as a data receiving end and the other side as a data transmitting end. Referring to the example of FIG. 6A, the controller 180 may control the second wireless communication terminal 20 to function as a data receiving end, and the first wireless communication terminal 10 may control to function as a data transmitting end. will be.

In order to control data communication between the first wireless communication terminal 10 and the second wireless communication terminal 20, the controller 180 is one of the first wireless communication terminal 10 and the second wireless communication terminal 20. The communication account of the communication counterpart may be transmitted to at least one. For example, by transmitting an account of the second wireless communication terminal 20 to the first wireless communication terminal 10, or by transmitting an account of the first wireless communication terminal 10 to the second wireless communication terminal 20, the first Data communication may be performed between the wireless communication terminal 10 and the second wireless communication terminal 20.

As described above, the communication account is a communication address (IP address and / or MAC address), telephone number, email address, SNS (Social Network Service) of the first wireless communication terminal 10 and the second wireless communication terminal 20. It may include at least one of identification (ID) and instant message (IM) service ID. The first wireless communication terminal 10 and the second wireless communication terminal 20 may perform data communication using any one of the plurality of communication accounts listed above.

In case of using the telephone numbers of the first wireless communication terminal 10 and / or the second wireless communication terminal 20, the first wireless communication terminal 10 and the second wireless communication terminal 20 use a text message service (SMS, LMS, MMS) may be used as a transmission means to perform data communication.

Otherwise, if an email address, an SNS ID, an IM service ID, or the like is used, the first wireless communication terminal 10 and the second wireless communication terminal 20 use data, e-mail, SNS, and IM services as transmission means for data communication. You can do it.

Further, when using a communication address such as an IP address, MAC address, etc., the first wireless communication terminal 10 and the second wireless communication terminal 20 communicate short-range communication technologies such as Zigbee, Bluetooth, Wi-Fi Direct, and the like. It will be possible to perform data communication. In detail, the first wireless communication terminal 10 and the second wireless communication terminal 20 may perform data communication by directly connecting to the counterpart terminal using a communication technology such as Zigbee, Bluetooth, Wi-Fi Direct, or the like.

When there are a plurality of communication accounts of the first wireless communication terminal 10 and / or the second wireless communication terminal 20, the controller 180 may control the communication means to be determined based on a user input.

For example, the controller 180 may determine a communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20 through the color of the first writing 601. For example, if the first color is an email, the second color is an IM service, the third color is a text message, and the fourth color is direct communication using short-range communication, the first wireless is based on the color of the first handwriting 601. Communication means between the communication terminal 10 and the second wireless communication terminal 20 can be determined.

For example, when the first handwriting 601 has a first color, the controller 180 allows the first wireless communication terminal 10 and the second wireless communication terminal 20 to perform data communication using an email as a communication means. If the first handwriting 601 has a second color, the controller 180 may control the first wireless communication terminal 10 and the second wireless communication terminal 20 to use the IM service as a communication means. It may be controlled to perform data communication. In addition to the color of the first writing 601, the communication means may be controlled to be determined based on at least one color of the second to ten writings to be described later.

As another example, when it is determined that the user's handwriting input is terminated, the controller 180 determines a communication means before starting data communication between the first wireless communication terminal 10 and the second wireless communication terminal 20. It may also display a pop-up window 610. As an example, FIG. 6B is an exemplary diagram of a pop-up window 610 being displayed for determining a data communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20.

The controller 180 may control the communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20 to be determined based on a user input to the pop-up window 610. For example, if a text message item is selected in the pop-up window 610 of FIG. 6B, the controller 180 determines that the first wireless communication terminal 10 and the second wireless communication terminal 20 are text messages (SMS, MMS, LMS). By using the communication means may be controlled to perform data communication.

According to another embodiment of the present invention, the controller 180 may control the communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20 to be determined based on the user's handwriting input. For example, in FIG. 6C, the controller 180 uses the communication means designated by the second handwriting 602 (illustrated as MMS in FIG. 6C) to the first wireless communication terminal 10 and the second wireless communication terminal 20. It is possible to control the data communication between the).

In performing data communication between the first wireless communication terminal 10 and the second wireless communication terminal 20, the controller 180 stores content stored in the first wireless communication terminal 10 in the second wireless communication terminal 20. ) May be controlled to be transmitted to the first wireless communication terminal 10 or the content stored in the second wireless communication terminal 20 may be transmitted to the first wireless communication terminal 10. In detail, the controller 180 may control the content stored in the wireless communication terminal functioning as the transmitting end to be transmitted to the wireless communication terminal functioning as the receiving end. In this case, the content transmitted from the transmitting end to the receiving end may be content output in real time through the wireless communication terminal functioning as the transmitting end. Here, the content may include a still image, a video, an e-book, a web page, a music file, a real-time output image, a text string for text message transmission, a phone book, and the like.

According to another embodiment of the present invention, the controller 180 may control the content specified by the user to be transmitted and received between the first wireless communication terminal 10 and the second wireless communication terminal 20. Referring to the exemplary diagram of FIG. 6D, the controller 180 is configured to display content designated by the third handwriting 603 (illustrated as '20120725.jpg' in FIG. 6D) from the first wireless communication terminal 10. 2 may be controlled to be transmitted to the wireless communication terminal 20. In detail, the controller 180 may control the '20120725.jpg' file stored in the first wireless communication module to be transmitted to the second wireless communication terminal 20.

6C and 6D illustrate that the communication means and the content between the first wireless communication terminal 10 and the second wireless communication terminal 20 are designated using the second handwriting 602 and the third handwriting 603. It became. When the user simultaneously inputs the second handwriting 602 and the third handwriting 603, the controller 180 may distinguish the second handwriting 602 and the third handwriting 603 by the separator 620. This will be described in detail with reference to FIG. 6E.

In FIG. 6E, a semicolon (;) is illustrated as the separator 620. The controller 180 may extract the second handwriting 602 and the third handwriting 603 based on the separator 620. That is, in the example of FIG. 6E, the controller 180 uses the second handwriting 602 written on the left side of the separator 620 as a communication means (illustrated as MMS in FIG. 6E), and the right side of the separator 620. Control so that the content designated by the third handwriting 603 written on the side (illustrated as a '20120725.jpg' file in FIG. 6E) is provided from the first wireless communication terminal 10 to the second wireless communication terminal 20. can do. As shown in the example of FIG. 6E, the separator 620 may be used as a means for separately recognizing each of a plurality of writings simultaneously input. Although FIG. 6E illustrates that the separator 620 is used between the communication means and the content, the purpose of the separator 620 is not limited thereto. For example, when a plurality of third handwriting 603 is input to provide a plurality of contents from the first wireless communication terminal 10 to the second wireless communication terminal 20, the plurality of third handwriting 603 is input. The separator 620 may be used to recognize the separation.

In FIG. 6E, the separator 620 is illustrated as a punctuation semicolon (;), but is not limited thereto. Math symbols, such as '+' and '-', as well as punctuation marks such as periods (.), Commas (,), colons (:), double quotes (“), single quotes (,), and exclamation points (!). Alternatively, conjunctions such as 'and', 'and', 'or', or 'or' may also be used as the separator 620.

If it is determined that the user's handwriting input is completed (S406), the mobile terminal 100 according to the present invention described with reference to FIG. 4 controls to start data communication between the first wireless communication terminal and the second wireless communication terminal. It was.

As a simple example, if a handwriting input is not applied to the touch screen 151 for a predetermined time after the input of one of the first to third handwriting ends, the controller 180 may determine that the user's handwriting input has ended. Can be. However, in this case, in order to control data communication between the first wireless communication terminal and the second wireless communication terminal, there may be a problem that a waiting time of a predetermined time occurs after the writing input.

Accordingly, when the drawing of the first handwriting in the form of a unidirectional arrow is completed, the mobile terminal 100 according to the present invention determines that the handwriting input of the user is completed, and thus, between the first wireless communication terminal and the second wireless communication terminal. It can be controlled to start data communication. In this case, after inputting the second handwriting and / or the third handwriting described above, the first handwriting may be completely input only after the first handwriting is input.

When the fourth handwriting 604 distinguished from the first to third handwriting is input, the mobile terminal 100 according to another embodiment of the present invention may determine that the user's handwriting input is finished. For example, as in the example illustrated in FIG. 6A, a fourth handwriting 604 that encloses the edge of the augmented reality object is input (in FIG. 6A, the frame of the 'smartphone' button, which is a virtual object, is audited. As illustrated in the example shown in FIG. 6C to FIG. 6E, when the fourth handwriting 604 in a form enclosing the second handwriting and the third handwriting is input, the controller ( 180 may determine that the user's handwriting input has ended.

The fourth handwriting 604 may be used to determine communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20. As described above, the controller 180 may control the communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20 to be determined based on the color of the fourth writing 604. . As another example, the controller 180 may control the communication means between the first wireless communication terminal 10 and the second wireless communication terminal 20 to be determined based on the form of the fourth handwriting 604. For example, when a circle is an email, a triangle is a text message, and a rectangle is assigned to an IM service, the controller 180 may control the communication means to be determined based on the form of the fourth handwriting 604. In the example shown in FIG. 6, since the fourth handwriting 604 is drawn in a circle, the control unit 180 uses the first wireless communication terminal 10 and the second wireless communication terminal 20 as the communication means for the data. It may be controlled to perform communication.

In FIG. 6, it is illustrated that both the first wireless communication terminal and the second wireless communication terminal are displayed in the mixed reality image. The mobile terminal 100 according to the present invention may control the wireless communication terminal not displayed on the mixed reality image to be a subject of data communication. This will be described in more detail with reference to FIG. 7.

FIG. 7 is an exemplary diagram for explaining control of data communication between a first wireless communication terminal displayed in the mixed reality image and a second wireless communication terminal not displayed in the mixed reality image. For convenience of description, the wireless communication terminal in the mixed reality image of FIG. 7 (illustrated as a smartphone in FIG. 7) is referred to as the first wireless communication terminal 10, but does not appear in the mixed reality image, but the first wireless communication terminal. A wireless communication terminal set to perform data communication with 10 will be referred to as a second wireless communication terminal.

The controller 180 may control to determine a second wireless communication terminal capable of performing data communication with the first wireless communication terminal 10 through a fifth writing 703 input. As illustrated in FIG. 7, when 'my notebook' is input as the fifth handwriting 703, the controller 180 controls the wireless communication terminal defined by the fifth handwriting 703 to function as a second wireless communication terminal. can do. That is, in the example of FIG. 7, a wireless communication terminal designated as 'My Notebook' may be designated as a second wireless communication terminal capable of performing data communication with the first wireless communication terminal 10. In the example of FIG. 7, since the first writing 701 input is a direction from the second wireless communication terminal ('my notebook') to the first wireless communication terminal 10 (smartphone), the data transmission direction is the second wireless. The communication terminal may have a form of transmitting data to the first wireless communication terminal 10.

As described above, when a fourth handwriting 702 having a form enclosing the augmented reality object of the fifth handwriting 703 or the first wireless communication terminal 10 is input (in FIG. 7, the form enclosing the fifth handwriting 703). The fourth handwriting 702 is illustrated as being inputted), and the user's handwriting input is determined to be finished, so that data communication can be controlled between the first wireless communication terminal 10 and the second wireless communication terminal. There will be.

6 and 7 illustrate that the controller 180 controls data communication between the first wireless communication terminal and the second wireless communication terminal. In addition, the controller 180 may control data communication between the first wireless communication terminal and the wireless communication unit 110 according to the first handwriting input and the second handwriting input. This will be described in detail with reference to FIG. 8.

8 is an exemplary diagram for describing data communication between a first wireless communication terminal and a wireless communication unit 110. For convenience of description, the wireless communication terminal in the mixed reality image of FIG. 8 (illustrated as a smartphone in FIG. 8) will be referred to as a first wireless communication terminal 10.

As in the example shown in FIG. 8, when the first handwriting 801 points to only one wireless communication terminal (first wireless communication terminal 10), the controller 180 is connected to the first wireless communication terminal 10. The data communication may be performed between the wireless communication units 110. In the example of FIG. 8, since the first handwriting 801 in the form of a unidirectional arrow is drawn to face the first wireless communication terminal 10, the wireless communication unit 110 is the data transmitting end, and the first wireless communication terminal 10 is the data receiving end. May be set to. 6B to 6E may also be applied as it is between the first wireless communication terminal 10 and the wireless communication unit 110. For example, as shown in FIG. 8B, a second handwriting 802 for designating communication means with the first wireless communication terminal 10 and a third handwriting 803 for designating transmission / reception content with the first wireless communication terminal 10. If is input, the content designated by the third handwriting 803 may be controlled to be transmitted and received by the communication means designated by the second handwriting 802. In the example of FIG. 8B, designation is made on the right side of the separator 810 by using the transmission means designated on the left side of the separator 810 (illustrated as a semicolon (;) in FIG. 8B) (illustrated as SMS in FIG. 8B). To send the content (example in the string 'come to the bulletin board in Figure 8b').

When data communication is performed between the wireless communication unit 110 and the first wireless communication terminal 10, the object of data communication is not necessarily limited to content. The controller 180 may remotely control an operation of the first wireless communication terminal 10 by controlling a control signal to be transmitted and received between the wireless communication unit 110 and the first wireless communication terminal 10.

As in the example shown in FIG. 8C, the first writing 801 is drawn to face the first wireless communication terminal 10, and the third writing 803 indicates the operating state of the first wireless communication terminal 10. If so, the controller 180 can transmit a control signal to the first wireless communication terminal 10 so that the first wireless communication terminal 10 can be set to an operation state designated by the third handwriting 803. In the example of FIG. 8C, since the operation state designated by the third handwriting 803 is 'power off', the controller 180 controls a control signal to turn off the power of the first wireless communication terminal 10. May be controlled to be transmitted to the first wireless communication terminal 10.

If it is determined that the user's handwriting input is terminated, the controller 180 may control data communication between the wireless communication unit 110 and the first wireless communication terminal 10. In the example of FIG. 8, it is illustrated that the end of the user input can be recognized by inputting the fourth handwriting 804 that surrounds the third handwriting 803.

6 to 8 illustrate that the reality object in the preview image is a wireless communication terminal having a communication account as additional information. Even if the reality object in the preview image is not a wireless communication terminal, if the reality object has a communication account as additional information, all of the details described above with reference to FIGS. 6 to 8 may be applied. For example, if the real object is a face of a specific counterpart and the additional information on the real object is a communication account of the specific counterpart, the controller 180 may control the wireless communication terminal of the communication counterpart to be a subject of data communication. When a face of a specific counterpart is displayed on the preview image, an operation of the mobile terminal 100 will be described with reference to the exemplary diagram of FIG. 9.

9 is an exemplary diagram for describing an operation of the mobile terminal 100 when the reality object is a face of a specific counterpart. When the face of the specific counterpart is recognized in the preview image, the controller 180 may control to display additional information 920 for the specific counterpart as shown in the example of FIG. 9.

Subsequently, when the first handwriting 901 is input to point to the augmented reality object of the specific counterpart (in the example of FIG. 9, the first handwriting 901 to point to additional information 920 ('Hong Gil-dong' of the specific counterpart)). ), The controller 180 may control the wireless communication terminal of a specific counterpart to be a subject of data communication. In the example of FIG. 9, when the fourth handwriting 904 is input, the controller 180 determines that the user's handwriting input is finished, and the communication means designated by the second handwriting 902 is illustrated as SMS in FIG. 9. ), The content designated by the third handwriting 903 (in FIG. 9, an example of a character string 'Come to the bulletin board') is transmitted.

6 to 9 illustrate that the transmitting end and the receiving end of the data communication are each set to one wireless communication terminal. The mobile terminal 100 according to the present invention may control the wireless communication unit 110, the first wireless communication unit 110, and / or the second wireless communication unit 110 to perform 1: N communication instead of 1: 1 communication. have. This will be described in detail with reference to the exemplary diagram of FIG. 10.

10 is an exemplary diagram for explaining that the wireless communication unit 110 is configured to communicate with a plurality of wireless communication terminals. For convenience of description, it is assumed that the persons illustrated in FIG. 10 are called first to fifth counterparts a-e, respectively, and the additional information on the first to fifth counterparts a-e is a communication account of each counterpart.

The user may input the sixth handwriting 1005 as in the example shown in FIG. 10 to select a plurality of wireless communication terminals to function as a transmitting end or a receiving end of the data communication. The controller 180 may control the wireless communication terminal of the counterpart included in the sixth handwriting 1005 to be a subject of data communication. In this case, in order to exclude at least one of the counterparts included in the sixth handwriting 1005, the user may input the seventh handwriting 1006 inside the sixth handwriting 1005 ('X in FIG. 10). '7th handwriting 1006' is illustrated as input). The controller 180 is a counterpart included in the sixth handwriting 1005 except for the counterpart excluded by the seventh handwriting 1006 (exemplified as the third counterpart c in FIG. 10) (FIG. 10). In FIG. 2, the wireless communication terminals of the first, second, and fourth counterparts (exemplified as a, b, and d) may be controlled to be subjects of data communication.

In the example of FIG. 10, when the fourth handwriting is input (1004), the controller 180 determines that the user's writing input has ended, and uses the communication means designated by the second handwriting 1002 to determine the third handwriting ( The content designated by 1003 may be controlled to be transmitted in the direction indicated by the first writing 1001.

Specifically, since the communication means of the MMS is designated as the second writing 1002 and the content file '20120725.jpg' is designated as the third writing 1003, the controller 180 controls the first, second, and fourth parties. The wireless communication terminal (a, b, d) may transmit the MMS with the file 20120725.jpg attached.

As in the example illustrated in FIG. 10, the controller 180 may control the wireless communication terminal included in the sixth handwriting 1005 to be a subject of data communication. At this time, if the seventh handwriting 1006 is included inside the sixth handwriting 1005, the wireless communication terminal defined by the seventh handwriting 1006 may be excluded from data communication.

In FIG. 10, a plurality of wireless communication terminals are set as data receiving ends. In addition to the example of FIG. 10, a plurality of wireless communication terminals may be set as data transmitting terminals.

4 to 10, the additional information about the real object is a communication account of the real object, and the controller 180 can control the real object to be a subject of data communication using the communication account of the real object. It was.

However, in the augmented reality environment, additional information about the reality object is not necessarily set to the communication account of the reality object. When the reality object in the preview image obtained from the camera 121 is a specific product, the additional information about the reality object may be a name of the product, a description of the product, price information of the product, information on a place of sale of the product, and the like. As such, if the additional information of the reality object is not a communication account, the reality object in the preview image may not be the subject of data communication.

When the additional information of the real object is not a communication account, the controller 180 may transmit the additional information of the real object to a specific counterpart based on a user's handwriting input. Detailed description thereof will be described in detail with reference to FIG. 11.

11 is an exemplary diagram for describing that additional information of a reality object is transmitted to a specific counterpart. For convenience of description, the reality object illustrated in FIG. 11 is a gift card, and the virtual object is illustrated as having additional information about the gift card (illustrated as 'detailed information' button 1110 in FIG. 11). .

As shown in the example of FIG. 11, when a user inputs the first handwriting 1101 and the eighth handwriting 1102, the controller 180 may output additional information indicated by the first handwriting 1101 to the eighth handwriting 1102. Can be sent to the communication account specified by). In the example of FIG. 11, one end of the first handwriting 1101 is in contact with the 'details' button 1110, and the eighth handwriting 1102 specifies a communication account of 'jeff.park@lge.com'. As such, the controller 180 may control the description of the gift card to be transmitted to the communication account jeff.park@lge.com.

In FIG. 11, the fourth handwriting 1104 described above is illustrated as being enclosed in the eighth handwriting 1102. When the fourth handwriting 1104 is input, the controller 180 determines that the inputs of the first handwriting 1101 and the eighth handwriting 1102 are finished, and the control unit 180 determines that the eighth handwriting 1102 is assigned to the communication account designated by the eighth handwriting 1102. The additional information indicated by the first handwriting 1101 may be controlled to be transmitted.

Mobile terminal 100 according to the present invention can be applied to the purchase of goods. The controller 180 may control the purchase button for the reality object in the preview image to be displayed as a virtual object. In this case, the controller 180 may control the purchase process for the reality object to be performed by the payment means determined by the user. This will be described in detail with reference to FIG. 12.

FIG. 12 is an exemplary diagram for describing an operation of the mobile terminal 100 that controls the purchase of a reality object using a virtual object in a reality mixed image. As in the example shown in FIG. 12, the controller 180 may control the purchase button for the reality object to be displayed as a virtual object (in FIG. 12, illustrated as a 'simple purchase' button 1210).

When the user inputs the first handwriting 1201 such that the purchase button touches one end, and then inputs the ninth handwriting 1203, the controller 180 purchases the real object using the payment means determined by the ninth handwriting 1203. You can control the procedure to be performed. In the example of FIG. 12, since the ninth handwriting 1203 is designated as 'my card', the controller 180 extracts card information corresponding to 'my card' from the memory 160 to purchase a real object. It can be controlled to be done. In FIG. 12, the card is designated as the payment means, but the present invention is not limited thereto, and the mobile phone micropayment or coupon may also be designated as the payment means.

In FIG. 12, the fourth handwriting 1204 described above is illustrated as being enclosed in the ninth handwriting 1203. When the fourth handwriting 1204 is input, the controller 180 determines that the input of the first handwriting 1201 and the ninth handwriting 1203 is finished, and is the payment means designated by the ninth handwriting 1203. The purchase procedure of the object may be controlled to be performed.

The mobile terminal 100 according to the present invention may control the visual effect to be applied to the reality object in the mixed reality image. Control to apply a visual effect to the reality object in the mixed reality image will be described in detail with reference to the embodiment of FIG. 13.

FIG. 13 is an exemplary diagram for describing a visual effect applied to a reality object in a mixed reality image. For convenience of description, the persons shown in FIG. 13 are referred to as first to third counterparts 1301 to 1303, respectively, and the virtual objects assigned to the first to third counterparts 1301 to 1303 are respectively referred to as first to third virtual persons. The object 11311-1313 will be referred to.

As in the example shown in FIG. 13A, when the user inputs the first handwriting 1321 and the tenth handwriting 1322, the control unit 180 performs the first handwriting 1321 as in the example shown in FIG. 13B. The visual object designated by the tenth handwriting 1322 may be reflected on the reality object designated by the user. In FIG. 10A, the visual effect designated by the tenth handwriting 1322 is 'mosaic', and the first handwriting 1321 points to the third virtual object 1313, so that the controller 180 may use the example shown in FIG. 10B. As described above, the face of the third counterpart 1303 corresponding to the third virtual object 1313 may be controlled to be displayed by being mosaicized.

If the first writing 1321 indicates the first virtual object 1311 or the second virtual object 1312, the controller 180 may face the first party 1301 or the second party 1302. The visual effects designated by the tenth handwriting 1322 may be controlled to be applied.

In FIG. 13, the fourth handwriting 1323 described above is illustrated as being enclosed in the tenth handwriting 1322. When the fourth handwriting 1323 is input, the controller 180 determines that the inputs of the first handwriting 1321 and the tenth handwriting 1322 have ended, and the controller 180 determines that the first handwriting 1321 is input to the reality object designated by the first writing 1321. 10 may be controlled to reflect the visual effect determined by the handwriting 1322.

According to one embodiment of the present invention, the above-described method (operation flow diagram) can be implemented as a code that can be read by a processor on a medium on which the program is recorded. Examples of the medium that can be read by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and may be implemented in the form of a carrier wave (e.g., transmission over the Internet) .

The above-described mobile terminal 100 is not limited in the configuration and method of the above-described embodiments, but the embodiments may be modified such that all or some of the embodiments are selectively combined .

100: mobile terminal
110:
121: camera
151: touch screen
160: Memory
180:

Claims (20)

Memory;
A wireless communication unit;
A camera for generating a preview image of the surrounding environment;
A touch screen for displaying a virtual object of the first wireless communication terminal in the preview image and the preview image; And
When the first handwriting is input to the touch screen,
A control unit for controlling data communication between a first wireless communication terminal indicated by the first handwriting and the wireless communication unit or a second wireless communication terminal between the first wireless communication terminal and the second wireless communication terminal;
. The method of claim 1,
The controller determines a data transmission direction in a direction designated by the first writing. 3. The method of claim 2,
And the first writing is a unidirectional arrow, and one end of the unidirectional arrow points to a virtual object of the first wireless communication terminal. The method of claim 3, wherein
If the other end of the one-way arrow points to the virtual object of the second wireless communication terminal,
And the controller controls the data communication to be performed between the first wireless communication terminal and the second wireless communication terminal. The method of claim 3, wherein
The mobile terminal of claim 1, wherein the one-sided arrow points to function as a data receiver and the other side to function as a data transmitter. The method of claim 1,
The control unit is a communication means designated by the second writing, and the mobile terminal, characterized in that for controlling to perform the data communication. The method of claim 1,
An object of the data communication between the first wireless communication terminal and the wireless communication unit is a content stored in the memory or the first wireless communication terminal or a control signal for controlling an operation of the first wireless communication terminal. Mobile terminal. The method of claim 7, wherein
And the controller controls the content designated by the third handwriting to be an object of the data communication, or controls the first wireless communication terminal to be set to an operation state designated by the third handwriting. The method of claim 1,
And a target of the data communication between the first wireless communication terminal and the second wireless communication terminal is content stored in the first wireless communication terminal or the second wireless communication terminal. The method of claim 9,
And the controller controls the content designated by the third handwriting to be an object of the data communication. The method of claim 1,
And the controller controls the communication means of the data communication to be determined through the color of the first handwriting. The method of claim 11,
The communication means is at least one of e-mail, text message, IM service, SNS and direct transmission using short-range communication. The apparatus of claim 1,
And if it is determined that the user's handwriting input is terminated, controlling the data communication to be started. 14. The apparatus of claim 13,
And when the first handwriting or the fourth handwriting is input, the user's handwriting input is determined to have ended. 14. The apparatus of claim 13,
And when it is determined that the user's handwriting input is terminated, the mobile terminal is configured to display a pop-up window for selecting a communication means of the data communication. The method of claim 1,
And the second wireless communication terminal is designated by a fifth handwriting. The method of claim 1,
And the control unit further controls to transmit additional information included in the virtual object indicated by the first handwriting to a communication account designated by the sixth handwriting. The method of claim 1,
And the control unit further controls to purchase an item corresponding to the virtual object indicated by the first writing by the payment means designated by the seventh writing. Generating a preview image of the surrounding environment;
Displaying a virtual object of the first wireless communication terminal in the preview image and the preview image;
Receiving a first handwriting input; And
Controlling data communication between the first wireless communication terminal and the wireless communication unit indicated by the first handwriting or between the first wireless communication terminal and the second wireless communication terminal.
And transmitting the control information to the mobile terminal. Instructions for generating a preview image of the surrounding environment;
Instructions for displaying the virtual object of the first wireless communication terminal in the preview image and the preview image;
Instructions for receiving a first handwriting input; And
Instructions for controlling data communication to be performed between the first wireless communication terminal and the wireless communication unit indicated by the first handwriting or between the first wireless communication terminal and the second wireless communication terminal.
And a control unit for controlling the operation of the mobile terminal.

Images