KR20180040955A - Mobile terminal and method for controlling the same - Google Patents

Abstract

Disclosed are a mobile terminal and a control method thereof. According to an embodiment of the present invention, the mobile terminal comprises: a touch screen capable of receiving a first input signal and a second input signal based on different input methods and outputting a first screen corresponding to a first application; and a control unit calling an input tool corresponding to the first input signal and displaying a graphic object indicating a corresponding input tool on the first screen if receiving the first input signal while the first screen is output. At this time, the control unit controls the touch screen to output a display corresponding to the second input signal by using an input tool corresponding to the graphic object based on an area where the graphic object is output if the second input signal is received while the first input signal is maintained and the graphic object is displayed. The present invention enables a user to easily call a desired input tool by using a hand based touch and a tool based touch at the same time, thereby enabling the user to perform more natural and intuitive handwriting input operations.

Description

[0001] MOBILE TERMINAL AND METHOD FOR CONTROLLING THE SAME [0002]

The present invention relates to a mobile terminal capable of performing an input using an input device and a control method thereof.

A terminal can be divided into a mobile terminal (mobile / portable terminal) and a stationary terminal according to whether the terminal can be moved. The mobile terminal can be divided into a handheld terminal and a vehicle mounted terminal according to whether the user can directly carry the mobile terminal.

The functions of mobile terminals are diversified. For example, there are data and voice communication, photographing and video shooting through a camera, voice recording, music file playback through a speaker system, and outputting an image or video on a display unit. Some terminals are equipped with an electronic game play function or a multimedia player function. In particular, modern mobile terminals can receive multicast signals that provide visual content such as broadcast and video or television programs.

Such a terminal has various functions, for example, in the form of a multimedia device having multiple functions such as photographing and photographing of a moving picture, reproduction of a music or video file, reception of a game and broadcasting, etc. .

As an example of such a function, in inputting a control command on the display of the terminal, the user can perform handwriting or the like using a tool such as a stylus pen without directly performing a touch with the finger.

If an input tool is required to perform input using the stylus pen, an operation for selecting a desired input tool with the stylus pen is performed, and an additional operation for returning to a previous state after using the selected input tool . At this time, while performing the operation for selecting the input tool and returning to the previous state, the input operation must be stopped. In particular, if the user has to use various input tools or if he / she has to perform input using a particular input tool frequently, it is inconvenient and troublesome to repeat the additional operation several times.

It is therefore an object of the present invention to provide a mobile terminal capable of easily and quickly calling, using, returning, and reusing an input tool without stopping writing when an input tool is required during handwriting using a stylus pen or the like And a control method thereof.

It is still another object of the present invention to provide a mobile terminal and a control method thereof that are more intuitive and improve the continuity of work when handwriting is performed using a tool such as a stylus pen.

Accordingly, a mobile terminal according to an embodiment of the present invention can receive a first input signal and a second input signal based on different input methods, and is capable of receiving a first screen corresponding to a first application, ; And a controller for calling the input tool corresponding to the first input signal when the first input signal is received while displaying the first screen and displaying the graphic object representing the input tool on the first screen And the control unit uses an input tool corresponding to the graphic object based on the area where the graphic object is output when the second input signal is received while the first input signal is maintained and the graphic object is displayed Thereby controlling the touch screen so that a display corresponding to the second input signal is output. Here, one of the first input signal and the second input signal is a hand-based input method, and the other is a pen-based input method.

In one embodiment, when the touch corresponding to the first input signal is released, the graphic object disappears from the first screen, and the display corresponding to the subsequent second input signal is displayed in the pre- And the control unit controls the touch screen so that the touch screen is output.

In one embodiment, the type of the input tool called on the basis of the first input signal is determined differently according to the touch method corresponding to the first input signal and the type of the first screen.

In one embodiment, when the touch corresponding to the first input signal is dragged while the graphic object is displayed, the control unit moves the graphic object along the dragged path, and uses the graphic object at the changed position So that a display corresponding to the second input signal is displayed.

In one embodiment, when the touch corresponding to the second input signal is dragged along the edge of the graphic object, the control unit controls the touch corresponding to the second input signal so that the display corresponding to the second input signal is aligned with the boundary line of the graphic object. And controls the output of the display corresponding to the two input signals while the touch corresponding to the second input signal is being dragged in the graphic object.

In one embodiment, if the graphics object is changed as the first input signal is maintained and a subsequent first input signal is received while the graphic object is being displayed, And outputs a display corresponding to the input signal.

In one embodiment, the control unit displays the graphic object while the first touch corresponding to the first input signal is maintained, and controls the second touch corresponding to the subsequent first input signal while maintaining the first touch. The input tool corresponding to the graphic object is changed differently.

In one embodiment, the control unit adjusts the magnification of the input tool corresponding to the first screen and the graphic object based on the degree of touch of the first touch corresponding to the first input signal.

In one embodiment, the control unit adjusts the scale of the input tool corresponding to the graphic object differently based on the direction and degree of the plurality of taps corresponding to the first input signal.

In one embodiment, the first screen includes at least one object, and when the second input signal is received in a state where a touch corresponding to the first input signal is applied to the object, And outputs a second graphic object to the first screen to fix a display and notify an input tool applicable to the object.

In one embodiment, when the second graphic signal is received and the subsequent second input signal is received while the second graphic object is being output and the first input signal is held, the control unit controls the touch object corresponding to the subsequent second input signal The attribute of the input tool corresponding to the second graphic object is mapped and the result of using the input tool corresponding to the subsequent second input signal is displayed on the object.

In one embodiment, while the first input signal is maintained, the other objects of the first screen are limited to not displaying the usage results of the input tool corresponding to the subsequent second input signal.

In one embodiment, when the touch corresponding to the first input signal is released, the control unit releases the attribute of the input tool mapped to the touch object, and if the touch corresponding to the first input signal causes the touch of the first screen And a third graphic object indicating the input tool applicable to another object is output to the first screen when the second graphic object is applied to another object.

In one embodiment, the first screen includes a handwritten input made using at least one of the first and second input signals, and the control unit is configured to: When the drag-touch input corresponding to the second input signal is applied to the area to which the handwriting input is output, the handwriting input is aligned so as to correspond to the drag trajectory of the drag-touch input while being displayed in the output area .

In one embodiment, the first screen includes at least one object, and when the object is selected using the first touch corresponding to the first input signal, And when a touch is applied to the first icon by using a second touch corresponding to the first input signal while maintaining the first touch, And when the second input signal is received, the execution result of the function is applied to the touch point at which the second input signal is received.

As described above, according to the embodiment of the present invention, a desired input tool can be easily called by performing a hand-based touch and a tool-based touch in parallel, and a more natural and intuitive handwriting input can be performed. In addition, it is possible to perform seamless and continuous tool-based handwriting input in parallel with the hand-based touch and the tool-based touch without repeatedly calling and returning the input tool each time the input tool is used. It also provides more intuitive interactivity by providing a UI like the use of real tools in parallel with hand-based touches and tool-based touches.

1A is a block diagram illustrating a mobile terminal according to the present invention.
FIGS. 1B and 1C are conceptual diagrams illustrating an example of a mobile terminal according to the present invention in different directions.
FIG. 1D is a diagram illustrating a touch input by a pen device as an example of an input method received on a touch screen of a mobile terminal according to the present invention.
FIG. 2 is a flowchart illustrating a method of inputting a touch screen of a mobile terminal according to an embodiment of the present invention.
FIG. 3 is a conceptual diagram illustrating an example of performing a more intuitive tool use input in parallel with a hand-based touch and a pen-based touch in a mobile terminal according to the present invention.
4, 5A, 5B and 6 are various examples of a method for changing the type and attribute of an applicable input tool using the degree of touch of the hand-based touch and the touch method in the mobile terminal according to the present invention.
FIGS. 7A, 7B, and 7C illustrate examples in which the use of an input tool is applied to a specific object in parallel with a hand-based touch and a pen-based touch in a mobile terminal according to the present invention.
Figs. 8, 9, 10A, 10B and 11 are diagrams for explaining a case where a hand-based touch and a pen-based touch are performed in parallel in the mobile terminal according to the present invention, Examples are shown.
12A, 12B, and 13 are conceptual diagrams illustrating examples of simple tool use inputs that can be performed using only a hand-based touch in a mobile terminal associated with the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or similar components are denoted by the same reference numerals, and redundant explanations thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The mobile terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC A tablet PC, an ultrabook, a wearable device such as a smartwatch, a smart glass, and a head mounted display (HMD). have.

The mobile terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC A tablet PC, an ultrabook, a wearable device such as a smartwatch, a smart glass, and a head mounted display (HMD). have.

However, it will be appreciated by those skilled in the art that the configuration according to the embodiments described herein may be applied to fixed terminals such as a digital TV, a desktop computer, a digital signage, and the like, will be.

1A to 1C are block diagrams for explaining a mobile terminal according to the present invention, and FIGS. 1B and 1C are conceptual diagrams showing an example of a mobile terminal according to the present invention in different directions.

The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, ), And the like. The components shown in FIG. 1A are not essential for implementing a mobile terminal, so that the mobile terminal described herein may have more or fewer components than the components listed above.

The wireless communication unit 110 may be connected between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and another mobile terminal 100 or between the mobile terminal 100 and the external server 100. [ Lt; RTI ID = 0.0 > wireless < / RTI > In addition, the wireless communication unit 110 may include one or more modules for connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115 .

The input unit 120 includes a camera 121 or an image input unit for inputting a video signal, a microphone 122 for inputting an audio signal, an audio input unit, a user input unit 123 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

The sensing unit 140 may include at least one sensor for sensing at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 226, a battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification can combine and utilize information sensed by at least two of the sensors.

The output unit 150 includes at least one of a display unit 151, an acoustic output unit 152, a haptic tip module 153, and a light output unit 154 to generate an output related to visual, auditory, can do. The display unit 151 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. The touch screen may function as a user input unit 123 that provides an input interface between the mobile terminal 100 and a user and may provide an output interface between the mobile terminal 100 and a user.

The interface unit 160 serves as a path to various types of external devices connected to the mobile terminal 100. The interface unit 160 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the mobile terminal 100, corresponding to the connection of the external device to the interface unit 160, it is possible to perform appropriate control related to the connected external device.

In addition, the memory 170 stores data supporting various functions of the mobile terminal 100. The memory 170 may store a plurality of application programs or applications running on the mobile terminal 100, data for operation of the mobile terminal 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Also, at least a part of these application programs may exist on the mobile terminal 100 from the time of shipment for the basic functions (e.g., telephone call receiving function, message receiving function, and calling function) of the mobile terminal 100. Meanwhile, the application program may be stored in the memory 170, installed on the mobile terminal 100, and may be operated by the control unit 180 to perform the operation (or function) of the mobile terminal.

In addition to the operations related to the application program, the control unit 180 typically controls the overall operation of the mobile terminal 100. The control unit 180 may process or process signals, data, information, and the like input or output through the above-mentioned components, or may drive an application program stored in the memory 170 to provide or process appropriate information or functions to the user.

In addition, the controller 180 may control at least some of the components illustrated in FIG. 1A in order to drive an application program stored in the memory 170. FIG. In addition, the controller 180 may operate at least two of the components included in the mobile terminal 100 in combination with each other for driving the application program.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to the components included in the mobile terminal 100. The power supply unit 190 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with one another to implement a method of operation, control, or control of a mobile terminal according to various embodiments described below. In addition, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 170. [

Hereinafter, the various components of the mobile terminal 100 will be described in detail with reference to FIG. 1A.

First, referring to the wireless communication unit 110, the broadcast receiving module 111 of the wireless communication unit 110 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast receiving modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The mobile communication module 112 may be a mobile communication module or a mobile communication module such as a mobile communication module or a mobile communication module that uses technology standards or a communication method (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution And an external terminal, or a server on a mobile communication network established according to a long term evolution (AR), a long term evolution (AR), or the like.

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be built in or externally attached to the mobile terminal 100. The wireless Internet module 113 is configured to transmit and receive a wireless signal in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 113 transmit and receive data according to at least one wireless Internet technology, including Internet technologies not listed above.

The wireless Internet module 113 for performing a wireless Internet connection through the mobile communication network can be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE or LTE- May be understood as a kind of the mobile communication module 112.

The short-range communication module 114 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB) (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short-range communication module 114 is connected to the mobile terminal 100 and the wireless communication system through the wireless area networks, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 ) And the other mobile terminal 100 (or the external server). The short-range wireless communication network may be a short-range wireless personal area network.

Here, the other mobile terminal 100 may be a wearable device (e.g., a smartwatch, a smart glass, etc.) capable of interchanging data with the mobile terminal 100 according to the present invention (smart glass), HMD (head mounted display)). The short range communication module 114 may detect (or recognize) a wearable device capable of communicating with the mobile terminal 100 around the mobile terminal 100. [ If the detected wearable device is a device authenticated to communicate with the mobile terminal 100 according to the present invention, the control unit 180 may transmit at least a part of the data processed by the mobile terminal 100 to the short- 114 to the wearable device. Therefore, the user of the wearable device can use the data processed by the mobile terminal 100 through the wearable device. For example, according to this, when a telephone is received in the mobile terminal 100, the user performs a telephone conversation via the wearable device, or when a message is received in the mobile terminal 100, It is possible to check the message.

The position information module 115 is a module for obtaining the position (or current position) of the mobile terminal, and a representative example thereof is a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the mobile terminal utilizes the GPS module, it can acquire the position of the mobile terminal by using a signal transmitted from the GPS satellite. As another example, when the mobile terminal utilizes the Wi-Fi module, it can acquire the position of the mobile terminal based on information of a wireless access point (AP) that transmits or receives the wireless signal with the Wi-Fi module. Optionally, the location information module 115 may perform any of the other functions of the wireless communication unit 110 to obtain data relating to the location of the mobile terminal, in addition or alternatively. The location information module 115 is a module used to obtain the location (or current location) of the mobile terminal, and is not limited to a module that directly calculates or obtains the location of the mobile terminal.

Next, the input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, Or a plurality of cameras 121 may be provided. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. [ A plurality of cameras 121 provided in the mobile terminal 100 may be arranged to have a matrix structure and various angles or foci may be provided to the mobile terminal 100 through the camera 121 having the matrix structure A plurality of pieces of image information can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or a running application program) being executed in the mobile terminal 100. Meanwhile, the microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 123 is for receiving information from a user and when the information is inputted through the user input unit 123, the control unit 180 can control the operation of the mobile terminal 100 to correspond to the input information . The user input unit 123 may include a mechanical input unit or a mechanical key such as a button located on a front or rear or side of the mobile terminal 100, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, And a touch key disposed on the touch panel. Meanwhile, the virtual key or the visual key can be displayed on a touch screen having various forms, for example, a graphic, a text, an icon, a video, As shown in FIG.

Meanwhile, the sensing unit 140 senses at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information, and generates a corresponding sensing signal. The control unit 180 may control the driving or operation of the mobile terminal 100 or may perform data processing, function or operation related to the application program installed in the mobile terminal 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 141 may be disposed in the inner area of the mobile terminal or in proximity to the touch screen, which is covered by the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection fluorescent light sensor, a mirror reflection fluorescent light sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 141 can be configured to detect the proximity of the object with a change of the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as "proximity touch & The act of actually touching an object on the screen is called a "contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 141 can detect a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, have. Meanwhile, the control unit 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Furthermore, the control unit 180 can control the mobile terminal 100 such that different operations or data (or information) are processed according to whether the touch to the same point on the touch screen is a proximity touch or a touch touch .

The touch sensor senses a touch (or touch input) applied to the touch screen (or the display unit 151) by using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, do.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched or the like. Here, the touch controller may be a separate component from the control unit 180, and may be the control unit 180 itself.

On the other hand, the control unit 180 may perform different controls or perform the same control according to the type of the touch object touching the touch screen (or a touch key provided on the touch screen). Whether to perform different controls or to perform the same control according to the type of the touch object may be determined according to the current state of the mobile terminal 100 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. Meanwhile, the controller 180 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 121 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

In addition, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 153 may be vibration. The intensity and pattern of the vibration generated in the haptic module 153 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 153 may synthesize and output different vibrations or sequentially output the vibrations.

In addition to vibration, the haptic module 153 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 153 can transmit the tactile effect through the direct contact, and the tactile effect can be felt by the user through the muscles of the finger or arm. The haptic module 153 may include two or more haptic modules 153 according to the configuration of the mobile terminal 100.

The light output unit 154 outputs a signal for notifying the occurrence of an event using the light of the light source of the mobile terminal 100. Examples of events that occur in the mobile terminal 100 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output from the light output unit 154 is implemented as the mobile terminal emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the mobile terminal detecting the event confirmation of the user.

The interface unit 160 serves as a path for communication with all external devices connected to the mobile terminal 100. The interface unit 160 receives data from an external device or supplies power to each component in the mobile terminal 100 or transmits data in the mobile terminal 100 to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 160.

The identification module is a chip for storing various information for authenticating the use right of the mobile terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM) A universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the interface unit 160. [

The interface unit 160 may be a path through which power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle, And various command signals may be transmitted to the mobile terminal 100. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The mobile terminal 100 may operate in association with a web storage that performs the storage function of the memory 170 on the Internet.

Meanwhile, as described above, the control unit 180 controls the operations related to the application program and the general operation of the mobile terminal 100. [ For example, when the state of the mobile terminal meets a set condition, the control unit 180 can execute or release a lock state for restricting input of a user's control command to applications.

In addition, the control unit 180 performs control and processing related to voice communication, data communication, video call, or the like, or performs pattern recognition processing to recognize handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the controller 180 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the mobile terminal 100 according to the present invention.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components. The power supply unit 190 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of an interface 160 through which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

Referring to FIGS. 1B and 1C, the disclosed mobile terminal 100 includes a bar-shaped terminal body. However, the present invention is not limited thereto and can be applied to various structures such as a folder type, a flip type, a slide type, a swing type, and a swivel type in which a watch type, a clip type, a glass type or two or more bodies are relatively movably coupled . A description of a particular type of mobile terminal, although relevant to a particular type of mobile terminal, is generally applicable to other types of mobile terminals.

Here, the terminal body can be understood as a concept of referring to the mobile terminal 100 as at least one aggregate.

The mobile terminal 100 includes a case (for example, a frame, a housing, a cover, and the like) that forms an appearance. As shown, the mobile terminal 100 may include a front case 101 and a rear case 102. Various electronic components are disposed in the inner space formed by the combination of the front case 101 and the rear case 102. At least one middle case may be additionally disposed between the front case 101 and the rear case 102.

A display unit 151 is disposed on a front surface of the terminal body to output information. The window 151a of the display unit 151 may be mounted on the front case 101 to form a front surface of the terminal body together with the front case 101. [

In some cases, electronic components may also be mounted on the rear case 102. Electronic parts that can be mounted on the rear case 102 include detachable batteries, an identification module, a memory card, and the like. In this case, a rear cover 103 for covering the mounted electronic components can be detachably coupled to the rear case 102. Therefore, when the rear cover 103 is separated from the rear case 102, the electronic parts mounted on the rear case 102 are exposed to the outside.

As shown, when the rear cover 103 is coupled to the rear case 102, a side portion of the rear case 102 can be exposed. In some cases, the rear case 102 may be completely covered by the rear cover 103 during the engagement. Meanwhile, the rear cover 103 may be provided with an opening for exposing the camera 121b and the sound output unit 152b to the outside.

These cases 101, 102 and 103 may be formed by injection molding of synthetic resin or may be formed of metal such as stainless steel (STS), aluminum (Al), titanium (Ti) or the like.

The mobile terminal 100 may be configured such that one case provides the internal space, unlike the above example in which a plurality of cases provide an internal space for accommodating various electronic components. In this case, a unibody mobile terminal 100 in which synthetic resin or metal is connected from the side to the rear side can be realized.

Meanwhile, the mobile terminal 100 may include a waterproof unit (not shown) for preventing water from penetrating into the terminal body. For example, the waterproof portion is provided between the window 151a and the front case 101, between the front case 101 and the rear case 102, or between the rear case 102 and the rear cover 103, And a waterproof member for sealing the inside space of the oven.

The mobile terminal 100 is provided with a display unit 151, first and second sound output units 152a and 152b, a proximity sensor 141, an illuminance sensor 142, a light output unit 154, Cameras 121a and 121b, first and second operation units 123b, a microphone 122, an interface unit 160, and the like.

1B and 1C, a display unit 151, a first sound output unit 152a, a proximity sensor 141, an illuminance sensor 142, an optical output unit (not shown) A second operation unit 123b, a microphone 122 and an interface unit 160 are disposed on a side surface of the terminal body, and the first camera 121a and the first operation unit And a mobile terminal 100 in which a second sound output unit 152b and a second camera 121b are arranged on the rear side will be described as an example.

However, these configurations are not limited to this arrangement. These configurations may be excluded or replaced as needed, or placed on different planes. For example, the first operation unit may not be provided on the front surface of the terminal body, and the second sound output unit 152b may be provided on the side of the terminal body rather than the rear surface of the terminal body.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

The display unit 151 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display display, a 3D display, and an e-ink display.

In addition, the display unit 151 may exist in two or more depending on the embodiment of the mobile terminal 100. In this case, the mobile terminal 100 may be provided with a plurality of display portions spaced apart from each other or disposed integrally with one another, or may be disposed on different surfaces, respectively.

The display unit 151 may include a touch sensor that senses a touch with respect to the display unit 151 so that a control command can be received by a touch method. When a touch is made to the display unit 151, the touch sensor senses the touch, and the control unit 180 generates a control command corresponding to the touch based on the touch. The content input by the touch method may be a letter or a number, an instruction in various modes, a menu item which can be designated, and the like.

The touch sensor may be a film having a touch pattern and disposed between the window 151a and a display (not shown) on the rear surface of the window 151a, or may be a metal wire . Alternatively, the touch sensor may be formed integrally with the display. For example, the touch sensor may be disposed on a substrate of the display or inside the display.

In this way, the display unit 151 can form a touch screen together with the touch sensor. In this case, the touch screen can function as a user input unit 123 (see FIG. 1A). In some cases, the touch screen may replace at least some of the functions of the first operation unit ().

The first sound output unit 152a may be implemented as a receiver for transmitting a call sound to a user's ear and the second sound output unit 152b may be implemented as a loud speaker for outputting various alarm sounds or multimedia playback sounds. ). ≪ / RTI >

The window 151a of the display unit 151 may be provided with an acoustic hole for emitting the sound generated from the first acoustic output unit 152a. However, the present invention is not limited to this, and the sound may be configured to be emitted along an assembly gap (for example, a gap between the window 151a and the front case 101) between the structures. In this case, the appearance of the mobile terminal 100 can be made more simple because the hole formed independently for the apparent acoustic output is hidden or hidden.

The optical output unit 154 is configured to output light for notifying the occurrence of an event. Examples of the event include a message reception, a call signal reception, a missed call, an alarm, a schedule notification, an email reception, and reception of information through an application. The control unit 180 may control the light output unit 154 to terminate the light output when the event confirmation of the user is detected.

The first camera 121a processes an image frame of a still image or a moving image obtained by the image sensor in the photographing mode or the video communication mode. The processed image frame can be displayed on the display unit 151 and can be stored in the memory 170. [

The first and second operation units 123b may be collectively referred to as a manipulating portion as an example of the user input portion 123 operated to receive a command for controlling the operation of the mobile terminal 100 . The first and second operation units 123b can be employed in any manner as long as the user touches, pushes, or scrolls in a tactile manner. Also, the first and second operation units 123b may be employed in a manner that the user operates without touching the user through a proximity touch, a hovering touch, or the like.

In the figure, the first operation unit () is a touch key, but the present invention is not limited thereto. For example, the first operation unit may be a mechanical key or a combination of a touch key and a touch key.

Contents input by the first and second operation units (123b) can be set variously. For example, the first operation unit receives commands such as menu, home key, cancellation, search, and the like, and the second operation unit 123b receives commands such as sound output from the first or second sound output units 152a and 152b And a switch to the touch recognition mode of the display unit 151 can be input.

On the other hand, a rear input unit (not shown) may be provided on the rear surface of the terminal body as another example of the user input unit 123. The rear input unit is operated to receive a command for controlling the operation of the mobile terminal 100, and input contents may be variously set. For example, commands such as power on / off, start, end, scrolling, and the like, the size adjustment of the sound output from the first and second sound output units 152a and 152b, And the like can be inputted. The rear input unit may be implemented as a touch input, a push input, or a combination thereof.

The rear input unit may be disposed so as to overlap with the front display unit 151 in the thickness direction of the terminal body. For example, the rear input unit may be disposed at the rear upper end of the terminal body such that when the user holds the terminal body with one hand, the rear input unit can be easily operated using the index finger. However, the present invention is not limited thereto, and the position of the rear input unit may be changed.

When a rear input unit is provided on the rear surface of the terminal body, a new type of user interface using the rear input unit can be realized. When the first operation unit is not disposed on the front surface of the terminal body in place of at least a part of the functions of the first operation unit provided on the front surface of the terminal body, The display unit 151 may be configured as a larger screen.

Meanwhile, the mobile terminal 100 may be provided with a fingerprint recognition sensor for recognizing the fingerprint of the user, and the controller 180 may use the fingerprint information sensed through the fingerprint recognition sensor as authentication means. The fingerprint recognition sensor may be embedded in the display unit 151 or the user input unit 123.

The microphone 122 is configured to receive the user's voice, other sounds, and the like. The microphone 122 may be provided at a plurality of locations to receive stereophonic sound.

The interface unit 160 is a path through which the mobile terminal 100 can be connected to an external device. For example, the interface unit 160 may include a connection terminal for connection with another device (for example, an earphone or an external speaker), a port for short-range communication (for example, an infrared port (IrDA Port), a Bluetooth port A wireless LAN port, or the like), or a power supply terminal for supplying power to the mobile terminal 100. The interface unit 160 may be implemented as a socket for receiving an external card such as a SIM (Subscriber Identification Module) or a UIM (User Identity Module) or a memory card for storing information.

And a second camera 121b may be disposed on a rear surface of the terminal body. In this case, the second camera 121b has a photographing direction which is substantially opposite to that of the first camera 121a.

The second camera 121b may include a plurality of lenses arranged along at least one line. The plurality of lenses may be arranged in a matrix form. Such a camera can be named an 'array camera'. When the second camera 121b is configured as an array camera, images can be taken in various ways using a plurality of lenses, and a better quality image can be obtained.

The flash 124 may be disposed adjacent to the second camera 121b. The flash 124 shines light toward the subject when the subject is photographed by the second camera 121b.

And a second sound output unit 152b may be additionally disposed in the terminal body. The second sound output unit 152b may implement a stereo function together with the first sound output unit 152a and may be used for implementing a speakerphone mode in a call.

The terminal body may be provided with at least one antenna for wireless communication. The antenna may be embedded in the terminal body or formed in the case. For example, an antenna constituting a part of the broadcast receiving module 111 (see FIG. 1A) may be configured to be able to be drawn out from the terminal body. Alternatively, the antenna may be formed in a film type and attached to the inner surface of the rear cover 103, or a case including a conductive material may be configured to function as an antenna.

The terminal body is provided with a power supply unit 190 (see FIG. 1A) for supplying power to the mobile terminal 100. The power supply unit 190 may include a battery 191 built in the terminal body or detachable from the outside of the terminal body.

The battery 191 may be configured to receive power through a power cable connected to the interface unit 160. In addition, the battery 191 may be configured to be wirelessly chargeable through a wireless charger. The wireless charging may be implemented by a magnetic induction method or a resonance method (magnetic resonance method).

The rear cover 103 is configured to be coupled to the rear case 102 so as to cover the battery 191 to restrict the release of the battery 191 and to protect the battery 191 from external impact and foreign matter . When the battery 191 is detachably attached to the terminal body, the rear cover 103 may be detachably coupled to the rear case 102.

The mobile terminal 100 may be provided with an accessory that protects the appearance or supports or expands the function of the mobile terminal 100. [ One example of such an accessory is a cover or pouch that covers or accommodates at least one side of the mobile terminal 100. [ The cover or pouch may be configured to interlock with the display unit 151 to expand the function of the mobile terminal 100. Another example of an accessory is a touch pen for supplementing or extending a touch input to the touch screen.

Hereinafter, embodiments related to a control method that can be implemented in a mobile terminal configured as above will be described with reference to the accompanying drawings. It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

The touch screen 151 of the mobile terminal 100 according to the embodiment of the present invention can distinguish input signals based on different input methods.

FIG. 1D shows touch input by the pen device 200 as an example of an input method received on the touch screen 151 of the mobile terminal according to the present invention.

Here, the touch screen 151 simultaneously performs an input function and a display function as described above. For this purpose, the touch screen 151 may include a display unit 211 and a sensing unit 212 as shown in FIG. 1D. The touch screen 151 may display various content screens through the display unit 211 and may be operated by the sensing unit 212 during the display of the content screen through the display unit 211, Based input signal, and can transmit the respective input signals to the control unit 180. [0050]

For this purpose, the sensing unit 212 may include a plurality of touch panels (not shown) for discriminating between touches (including proximity touch) by a user's hand and touches (including proximity touch) May be combined or integrally formed. In addition, the plurality of touch panels can sense the touch by any one of a capacitive type, a resistive type, an ultrasonic type, an infrared type, and an electromagnetic induction type according to the type of the pen device 200. The sensing unit 212 may detect the coordinate where the input signal is generated when a hand-based input signal on the touch screen 151 or an input signal based on the touch of the pen device 200 is detected, And transmit the detected coordinates to the control unit 180. [ Then, the control unit 180 may perform a function corresponding to the received input signal.

In addition, the pen device 200 of FIG. 1D may be a pen-type input tool capable of performing input to the touch screen 151. The pen device 200 may use an electromagnetic resonance method using an LC resonance circuit or a passive method using a delay element, but the present invention is not limited thereto. The body of the pen device 200 may include a button capable of transmitting an input signal to the controller 180 of the mobile terminal 100 so as to perform a specific function. Here, although the pen device 200 is shown as an independent device from the mobile terminal 100, the frame of the mobile terminal 100 may be removably attached. In the pen device 200, a magnetic sensor and an acceleration sensor included in the mobile terminal 100, including a plurality of magnetic field generators, for example, dipole magnets, The distance from the touch screen 151, the position on the space outside the touch screen 151, and the like.

The sensing unit 140 senses a change in a current received from the pen device 200, a change in an electromagnetic wave, or a pressure change of the pen device 200 input to the touch screen 151, have. The control unit 180 controls the inclination of the pen device 200 and the position of the pen device 200 projected on the touch screen 151 and the position of the pen device 200 based on the information received from the sensing unit 140. [ The distance between the pen device 200 and the touch screen 151, and the like.

In addition, the touch screen 151 according to the present invention can simultaneously receive an input signal by a hand touch and an input signal by a touch of the pen device 200, and generates an input signal differentiated from each other and transmits the input signal to the control unit 180 . At this time, when an input signal by hand touch and an input signal by touching the pen device 200 are sequentially received, a control command corresponding to a later received input signal is affected by the input signal received first . That is, the control unit 180 can generate a different control command by distinguishing the input signal of the hand-touch input and the input signal of the pen unit 200, thereby providing more various input methods. Thus, in the present invention, the input by the hand touch and the input by the touch of the pen device 200 interact, thereby providing a more natural and intuitive input method.

Hereinafter, FIG. 2 is a representative flowchart illustrating an operation for providing a new input method in parallel with a different input method on a touch screen of a mobile terminal according to the present invention.

Referring to FIG. 2, a first screen may be displayed on the touch screen 151 of the mobile terminal 100 (S10). Here, the first screen may include any type of screen capable of generating control commands such as data, text, image input, modification, composition, call, edit, update, transmission, etc. based on user input have. For example, the first application may be a memo application and the first screen may be a new / saved page corresponding to the execution of the memo application. Further, the first screen is not limited to the execution screen of the specific application. Accordingly, when the touch screen 151 is in a disabled state and the screen is not output, a capture screen of a home screen may be included.

When the first input signal based on one of the hand touch method and the touch method by the pen device is received while the first screen is being output, the controller of the mobile terminal 100 responds to the received first input signal You can call the input tool.

At this time, the type of the input tool to be called may vary depending on the type of the first screen output to the touch screen 151 and the touch method corresponding to the first input signal applied to the touch screen 151. To this end, the controller 180 recognizes whether the input method of the received input signal is the hand-based touch or the touch based on the pen device 200, thereby determining what type of tool to select.

For example, when the first screen is the execution screen of the memo application and the touch corresponding to the first input signal is the long-touch inputs applied to the two touch points as the hand-based touch, a 'ruler' Can be called. In addition, for example, if the touch corresponding to the first input signal is a touch method based on the pen device 200, a default toolbox may be called.

At this time, even if a touch corresponding to the first input signal is recognized as a touch method by a hand, another tool may be called according to a specific touch input method. Specifically, different tools connected depending on the number of touch points, the time of holding the touch, the degree of touch, the degree and direction of the drag, and the like may be called. To this end, the memory 170 of the mobile terminal 100 may store various touch input methods and information about matching input tools. Also, the memory 170 may include various functions corresponding to combinations of the first and second input signals input through different input methods, and these functions may be provided to the designer of the application or the user of the mobile terminal 100 Can be set by

When the input tool is called in this way, a graphic object representing the called input tool is output to one area of the first screen (S30). Here, the graphic object may correspond to the image of the called input tool, that is, the shape, size, color, etc. of the input tool. In addition, the position of the graphic object may correspond to a touch point of the touch corresponding to the first input signal.

Then, while the first input signal is retained and the graphic object is displayed, an input method other than the input method corresponding to the first input signal (e.g., a touch based on a hand-based touch or a pen device) When the second input signal based on the touch of the touch screen or the touch of the hand is received, the control unit controls the second input signal by using the input tool corresponding to the graphic object, based on the area where the graphic object is output The screen can be controlled so that a corresponding display is output (S40).

Specifically, the control unit holds or fixes the called input tool on the screen while the touch input of the first input method corresponding to the first input signal is maintained. When the touch input of the second input method corresponding to the second input signal different from the first input method is applied, the control unit performs input using the input tool based on the position where the input tool is held, You can perform the input without tools.

For example, when 'ruler' is called as an input tool, the user draws a curved line on the screen using the pen device 200 with the other hand held in a hand, A ruler can be used to draw a straight line from time to time without interrupting the operation.

According to this, when a user draws a curve and a straight line on the execution screen of the Notepad application several times, it is not necessary to repeatedly call up and cancel the necessary tool, so seamless input is possible without interruption Do. As a result, the continuity of the input operation is further improved.

Fig. 3 is a conceptual diagram showing an example of performing a more intuitive tool use input in parallel with a touch by a hand and a touch by a pen device.

3, a new page 301 corresponding to the execution of the memo application is outputted to the touch screen 151 of the mobile terminal 100, and a new page 301 corresponding to the execution of the memo application is displayed on the touch screen 151 of the mobile terminal 100, Is applied, a ruler may be called with an input tool matching therewith.

The tool box 310 related to the memo input may be popped up at the lower end of the page 301 and the ruler image 311 in the popped up tool box 310 may be displayed at the center of the touch screen 151 As shown in FIG. At the same time, the child image 311 is an indication that it has been selected as an input tool. For example, a highlighting effect may be output at the boundary of the child image 311.

Thereafter, when the predetermined time has elapsed, the graphic object 320 representing the child image may be output based on the position of the plurality of touch points FT applied to the touch screen 151. [ At this time, the graphic object 320 may be a shape of an actual character, for example, a rectangular bar-shaped rectangular image. The position of the outputted graphic object 320 can be moved as the position of the plurality of touch points FT is changed. If there is a memo input in the area where the graphic object 320 is output, by adjusting the transparency value in the corresponding area in the graphic object 320 so that the memo input is not blocked by the graphic object 320, It is possible to identify the memo input.

On the other hand, when the graphic object 320 is output and the touch input corresponding to the plurality of touch points FT is maintained, the user performs a yaw operation or a pivot operation based on the plurality of touch points FT pivot) operation, the graphic object 320 may move or rotate according to the direction of the operation. At this time, an icon 330 indicating the degree of movement or rotation may be displayed in the graphic object 320, for example. For example, the icon 330 may be displayed with a rotation angle of 45 degrees.

In addition, although not shown, the control unit may add or replace the input tool based on a predetermined touch gesture of the hand-based method corresponding to the first input signal.

In this manner, when a touch based on the pen device 200, rather than a hand-based touch, is received while the touch input corresponding to the plurality of touch points FT and the display of the graphic object 320 of the character image are maintained, The control unit can distinguish it from the input based on the hand-based touch, and can generate the control command corresponding to the touch based on the pen device 200. [

At this time, if the touch point of the touch based on the pen device 200 is within or near the boundary of the area where the graphic object 320 is output, the control unit outputs the input display using a tool corresponding to the graphic object 320 can do. 3, when a drag-touch input is applied in the vicinity of the boundary line of the graphic object 320 using the pen device 200, a straight line 302a, which is a line drawing a ruler, do. That is, even if the drag-touch input does not exactly follow the boundary of the graphic object 320, a straight line that matches the edge of the graphic object 320 is drawn. At this time, even if the position of the plurality of touch points FT changes in a predetermined range due to a small hand tremor of the user, the control unit can display the graphic object 320 while performing the straight drawing input through the pen unit 200. [ It is possible to fix the position of the optical disc 100 in a continuous manner.

In addition, when the pen device 200 attempts to enter the graphic object 320 in the vicinity of the boundary of the graphic object 320, it is possible to limit the output of the display corresponding to the drag touch input, have. At the same time, the preset vibration sound can be output a predetermined number of times.

If the touch point of the touch based on the pen device 200 is far away from the area where the graphic object 320 is output even while the graphic object 320 is being displayed, You can output a line of free form that follows. In this case, when the user brings the plurality of touch points FT to the position where the pen device 200 is located, the input using the ruler tool, that is, the straight drawing, can be successively performed as described above.

Next, when the touch input corresponding to the plurality of touch points FT applied to the touch screen 151 is released, the graphic object 320 representing the called hand tool disappears and the free form drawn by the pen device 200 disappears The line 302b of FIG. To this end, when the touch corresponding to the first input signal, for example, the hand-based touch, is released, the control unit erases the graphic object 320 representing the called input tool, The touch screen 151 can be controlled so that the touch screen 151 is output as the pre-call attribute value of the input tool.

As described above, according to the present invention, necessary input tools are fetched by using the hand-based touch, and then parallel input using the input tool called using the free input and the hand-based touch using the pen- It is possible without. Accordingly, it is possible to improve the continuity of the work in performing the input operation using the tool by the user.

It also provides a more intuitive and realistic metaphor sensibility in the way tool inputs are used.

Figure 4 shows an example of how to change the called input tool.

First, as described above, the type of the input tool called on the basis of the first input signal, for example, the touch signal generated by the touch based on the hand-based touch or the pen device 200, Method and the type of the first screen.

Here, if at least one of the number of touches, the number of touches, the holding time of touches, and the touch degree is different, it can be said that the touch method is different. It can be said that the type of the first screen is different if the type of the executed application is different, whether the same application has different execution steps, or whether the data inputted on the first screen is different. For example, a ruler can be called with an input tool if the number of touches is two. Also, for example, when a single long touch input is applied, an eraser may be called as an input tool. In addition, when the web page is outputted to the first screen, the area capturing tool can be called with the input tool, and when a new memo page is outputted to the first screen, a ruler or a protractor can be called. Also, which input tool is matched to which touch gesture may be implemented in a form that is set or changed through user input.

Fig. 4 is an example of changing the input tool in such a state that the input tool is called and displayed.

4, a plurality of taps corresponding to the hand-based touch are applied to the touch screen 151, thereby confirming that the character image 410 is output. At this time, when the user applies the flicking touch input downward by using a finger other than the thumb and the stop corresponding to the plurality of touches, for example, the index finger, the control unit can change the called input tool to another input tool have. Accordingly, it can be seen that the child image 410 is changed to the protractor image 420 as shown in FIG. At this time, the touch by the thumb and the stop finger must be maintained while the protractor image 420 is being output.

When the input tool is changed in this way, the changed input tool's attributes are applied to the display of the input by another input method thereafter. 4, the pen device 200 is brought close to the self-image 410, and when the self-image 410 is drawn, a straight line is output. In this state, the self- 420), the scales corresponding to the touch start point and the touch end point that the pen device 200 touches can be immediately displayed (e.g., highlighted) in the scale area of the protractor image 420.

In this state where the input tool has been changed to the protractor, the control unit again inputs the previous input tool, i.e., the ruler, in the direction opposite to the first, for example, And display it between a plurality of touch points.

In addition, memory 170 may store information related to the input tool (e.g., the calling order of the input tool) that matches the direction and frequency of such flicking touch input, and this information may be changed based on user input .

Next, FIGS. 5A, 5B and 6 show examples in which only the attribute of the input tool is changed while keeping the called input tool using the touch corresponding to the first input signal.

Referring to FIG. 5A, when an image 501a is drawn on the touch screen 151 using the pen device 200 and a straight line is required to be drawn, the user uses the hand-based touch (f) Can be called. Accordingly, when the user applies a touch to the hand-based touch f by further applying the touch (ff) while the character image 510a is displayed on the screen, the control unit displays the character image 510a and the user- ) Can be adjusted in proportion to the degree of touch. For example, the magnification of the user-drawn image 501a and the ruled-line image 510a is increased (501b, 510b) based on at least the area to which the touch point is applied, so that more detailed drawing and drawing can be performed. Conversely, by reducing the pressure of the hand-based touch ff, the magnification of the magnification-increased image 501b and the magnification of the magnetic image 510b can be reduced at the same time.

At this time, when the pen device 200 is touched or hovered at an arbitrary point on the touch screen 151 to start the input, even if there is a slight change in the pressure of the hand-based touch, It is possible to stop the magnification control for a predetermined time. That is, the magnification is fixed for a predetermined time even if there is a change in the pressure of the hand-based touch so that the user does not disturb the drawing with the pen device 200.

In a similar example, FIG. 5B shows a case where the touch (ff) is applied by further applying the hand-based touch f while the protractor image 520a is being displayed. Then, the scales displayed in the protractor image 520a are further segmented and the image 502a drawn by the user is also increased in proportion to the touch pressure. As a result, the user is immediately provided with more detailed angular information in drawing.

5A and 5B, the graphic objects 510b and 520b representing the input tool are displayed at the original magnification (510a and 520a) for a predetermined time, When the time has elapsed, it may disappear from the screen. At this time, the magnification of the drawn image may be maintained while the touch input by the pen device 200 is maintained. That is, even if the input tool disappears as the hand-based touch is released, the enlarged magnification of the picture image can be maintained for finer work until the touch input by the pen device 200 is released.

Next, FIG. 6 shows an example in which only the scale of the attributes of the input tool is changed while maintaining the magnification of the screen.

6, when a hand-based touch is applied to two points of the touch screen 151 in a state that a drawn image 601 is outputted through the pen device 200 to the touch screen 151, A ruler image 610a may be output. When the ruled-line image 610a is output as described above, a grid paper image may be displayed on the drawing image 601 so that the scale unit (centimeter, 10mm) displayed on the ruled-line image 610a can be easily distinguished.

At this time, if a pinch-out touch gesture is dragged in the direction in which the two points are moved away from each other, the scale unit displayed on the child image 610a may be changed to a larger unit (610b). For example, the scale unit of a ruler can be changed from centimeters (10 mm) to inches (25 mm). As a result, the grid size of the grid paper image displayed on the drawing image 601 also increases. At this time, a graphic object indicating a changed scale unit, for example, a bar-shaped indicator, may be output at a position adjacent to the child image 610b. However, unlike FIGS. 5A and 5B, the magnification of the picture image 601 maintains the previous state.

Similarly, if a pinch-in-touch gesture is performed in which the two points in the child image 610b are dragged in the direction of approaching each other, the scale unit of the ruler can be reduced again. In this case, the grid size of the grid paper image displayed on the drawing image 601 is reduced again.

Also, in one embodiment, if a plurality of taps are applied to the input tool image, it may be implemented to immediately return to the original scale unit.

Next, FIGS. 7A, 7B, and 7C illustrate examples of a method of applying the use of the input tool only to a specific object on the screen, in parallel with the hand-based touch and the pen device-based touch.

Here, it is assumed that the first screen output to the touch screen 151 includes at least one object. Here, the type of the object is not limited. For example, the object may include a text, an image, and a moving picture. In addition, the object may include a menu, an icon, and an application icon for executing a specific function.

Referring to FIG. 7A, a screen 701 including a plurality of objects may be displayed on the touch screen 151. FIG. Here, the plurality of objects may be various graphic images 711, 712, and 713. At this time, when the hand-based touch is applied to any one of the graphic images 712, the control unit recognizes that the corresponding graphic image 712 is selected as an object to be edited.

Thereafter, when the pen apparatus 200 is close to the selected figure image 712, the control unit recognizes it as a hovering state and outputs the tool box 720 associated with the figure image 712 in a contiguous manner. Here, the tool box 720 includes only the input tools related to the selected figure image 712. [ Thus, when another figure image 711 or 713 is selected, a tool box containing other input tools may be popped up. The tool box 720 is used to provide only input tools related to the selected object. When the pen device 200 is hovered at an arbitrary point on the touch screen 151, Is different.

Thus, while the pen device 200 is close to the figure image 712 and the tool box 720 is being displayed, the position of the figure image 712 is still fixed. That is, the hand-based touch applied to the graphic image 712 plays a role of fixing the object like a pin tool. Accordingly, when the touch screen 151 of the mobile terminal 100 is switched from the portrait mode to the landscape mode of Fig. 7A, even if the other figure images 711 and 713 rotate along the screen direction in which the change is made, The switching of the screen direction may not be applied to only the graphic image 712.

When the specific tool included in the tool box 721 is selected using the pen device 200, the control unit of the mobile terminal 100 displays the attributes of the selected specific tool, for example, the rotation tool 721 And maps it to the pen device 200. Accordingly, a wheel region 730 for adjusting the degree of rotation by the pen device 200 can be popped up on the screen 701. [

Then, the control unit can control the input corresponding to the subsequent second input signal, for example, the result of using the rotating tool corresponding to the touch input based on the pen device 200 to appear on the selected object. 7A, when the pen device 200 performs a rotation in the wheel region 730 or performs a rotation in an arbitrary region, the wheel region 730 has an animation effect rotating in the corresponding direction and degree Can be output. Then, the figure-based image 712 to which the hand-based touch is applied is rotated (712a) corresponding to the display output to the wheel region 730.

Meanwhile, while the rotation tool is applied to the hand-touched graphic image (712a), the use effect of the rotation tool is not applied to the other objects (711, 713) of the screen (701). That is, the use effect of the rotating tool is applied only to the specific object 712. In addition, according to this, it is possible to apply a desired degree of rotation effect more easily and quickly than when the specific object 712 is directly rotated using only the touch based on the hand base or the pen device 200. [

When the hand-based touch applied to the selected object 712 is released, the control unit can release the attribute of the input tool, i.e., the rotating tool, that has been mapped to the pen device 200. [ As a result, the popped-up wheel region 730 also disappears. If a hand-based touch is applied to another object 711 or 713 of the screen 701, the control unit may output another tool box to the screen 701 indicating the input tool applicable to the selected other object.

7A, when the hand touch applied to the figure image 712 is moved to another object 711 or 713 in the state where the wheel region 730 is popped up, the effect of the rotating tool is applied to another selected object If it can, it can be immediately applied to that object. At this time, the effect of using the rotating tool is no longer displayed in the figure image 712.

Next, FIG. 7B shows a case where a cutting tool 722 is selected in the tool box 720 of FIG. 7A. In this case, the pen attribute is mapped to the pen device 200. Accordingly, when the pen device 200 passes over the figure image 712 in which the hand touch is maintained, an image effect in which the figure image 712 is cut along the drag path is output.

Thereafter, when the pen device 200 passes the figure image 712 as shown in Fig. 7B, the figure image 712 is divided into the first part 712b and the second part 712c as if scissors were cut. At this time, the image effect that the pen device 200 cuts off even though it passes the other objects 711, 713 of the screen 701 is not applied. That is, the selected tool application effect appears only for the object in which the hand touch is maintained.

In addition, when a short-touch is applied to an arbitrary area to which the other objects have not been output after the pen device 200 completely passes the figure image 712 as shown in FIG. 7B, the hand touch applied to the figure image 712 is released The mapping tool is unmapped as it was. On the other hand, if a short-touch is applied to an arbitrary area where no other objects are output before the pen device 200 completely passes through the figure image 712 (i.e., after cutting a little) The cut-out portion of the cutting tool 712 may be returned to the original state, and the mapping of the cutting tool may be canceled.

Next, FIG. 7C shows a case where an outline tool is selected in the tool box 720 of FIG. 7A. In this case, if the pen device 200 to which the attribute of the outline tool is mapped is dragged along the edge of the figure image 712, the control unit displays the display of the figure image 712 by the pen device 200 You can control to appear only outside the boundary line. At this time, the control unit can restrict the display by the touch while the touch position of the pen device 200 is on the graphic image 712. That is, the screen image 701 can be controlled such that the graphic image 712 functions as a masking tape.

On the other hand, in the other figure images 711 and 713, the input by the pen device 200 can be displayed as it is.

Next, Figs. 8, 9, 10A, 10B, and 11 show a more convenient and intuitive tool use input in parallel with the touch by the hand and the touch by the pen device in the mobile terminal related to the present invention As well as other examples.

First, Fig. 8 shows an example of easily changing the arrangement of handwriting input performed on the touch screen 151. Fig. To this end, as shown in FIG. 8, a handwriting input through a user input may be output to the screen 801 output to the touch screen 151. At this time, the handwriting input may be hand-based or by the pen device 200.

8, in the state where the hand tool is called by using the hand-based touch, the hand image based drag-touch input can be used to move the character image 810 to the area 802 where the handwriting input is output (810M ). If the handwriting is kept such that the character image 810 is placed under the area 802 where the handwriting input (e.g., 'Hello?') Is output, if a straight line underwriting is made to the handwriting input using the pen unit 200, The handwriting input which has been outputted in a jagged manner corresponds to the straight line underline, that is, to correspond to the drag trajectory of the pen device 200. [ Thereafter, when the hand touch and the touch by the pen device 200 are released, only the aligned handwriting input 801A is left.

In addition, although not shown, the invoked input tool using the hand touch with the pen device 200 may be used to interactively change the format of the handwriting input, such as paragraphs, line breaks, or text color changes . At this time, the format information of the handwriting input and the guide information related to the color change can be output adjacent to the image representing the called input tool.

Next, FIG. 9 shows an example of easily copying and pasting the same image as the green image using the pen device 200.

(Hereinafter referred to as a "first point") of the area 903 in which the original image is output, after the original image 902 is drawn using the pen device 200 on the memo screen 901 shown in FIG. 9, Based touches are applied to an arbitrary point (hereinafter, referred to as a 'second point') out of the first image 902 and the region 903, the controller performs copying as an example of the selected original image 902 or an input tool related to the region 903 The icon 910 can be generated. At this time, the icon 910 may be positioned between the first point and the second point so that the hand-based touch is easy.

For example, when the user touches the icon 910 using the index finger, the copy result of the original image 902 may be output 904 to the designated area, for example, the upper left of the screen 901. Thereafter, when the pen device 200 touches a desired point (hereinafter, the 'third point') of the touch screen 151, the original image 904 is immediately And moved to the third point (904M). That is, the control unit can immediately apply the execution result of the icon 910 to the point where the touch based on the pen device 200 is applied. Thereafter, the user can complete a desired picture image with only the pen device 200 (904).

According to this, the user can use the input tool only by using the hand touch at that time, and the pen device 200 has an advantage of seamlessly continuing the desired work. That is, the user can concentrate only on the drawing of the snowman image 904 with the pen device 200.

10A and 10B are for solving the inconvenience that may occur when a touch input based on the hand-based or pen device 200 is performed at the edge of the touch screen 151 . The mobile terminal 100 according to the present invention may not include a user input unit such as a back button and / or a home button on a front surface of a main body provided with a touch screen 151. [ Accordingly, when an input is applied to the innermost surface of the touch screen 151 (BT), a back control command can be executed as shown in FIG. 10A, whereby the current screen 1001 is unintentionally moved It can be switched to the screen 1002 that has been output.

10B, when a plurality of hand-based touch inputs pushed to the right side of the touch screen 151 are applied, a lock image 1010 for restricting input of a back control command is outputted . At this time, the screen currently output is displayed in the first area 1001 of the touch screen 151 in a slightly pressed state to the right, and only the lock image 1010 is output to the null area in the second area 1003 .

At this time, when the user holds the hand-based touch inputs A or releases the touch inputs A after touching the lock image 1010, the left side of the first area 1001 through the pen device 200 Input can be done to the end. When the input is completed, releasing the hand-based touch inputs A or touching the lock image 1010 again, the first area 1001 is pulled to the right like an elastic string, and the second area 1003 disappears. Accordingly, it is possible to perform handwriting input to the left end of the current screen without fear of screen switching.

11 illustrates an example in which a touch based on the hand-based touch and a touch based on the pen device 200 interacts with a document 1101 to which information is input on the touch screen 151. FIG.

In FIG. 11, when touch inputs are applied to the first and second hand-based points, an icon 1110 for creating a highlighting area having both the first point and the second point may be popped up nearby. A 'Highlight' indication may be displayed on the icon 1110 to guide the function to be executed. When a touch is applied to the icon 1110, a highlighting area 1120 having both ends of the first point and the second point is output, and the associated tool bar box 1130 may be output at the lower end of the screen 1101.

At this time, if the pen device 200 is applied to the highlighting area 1120, the highlighting area 1120 can be finely adjusted in line units 1120M. In this case, the highlighting area 1120 may be initially designated using the hand-based touch, and then the start line and the end line may be finely adjusted using the pen device 200.

Next, Figs. 12A, 12B, and 13 are conceptual diagrams showing another example of simply using the tool using only the hand-based touch.

12A, if the user does not have a user input unit on the front surface of the mobile terminal 100, the touch gesture A, which is a hand-based pinch, is applied to the current screen 1201, The screen state 1202 (hereinafter, referred to as 'second screen') for returning to the screen of the previous step. At this time, in a state in which the hand-based touch is not released (at this time, a Move icon 1210 for guiding screen movement as shown in FIG. 12A may be displayed), and another finger not touched, When a flicking touch input is performed using a finger (B1), the user can move to the upper steps (P, H) (or lower steps) of the current screen 1204 at one time.

12A, when a hand-based pinch-out touch gesture A is applied on the second screen 1202, it is recognized that the screen step in the center has been selected, so that the user exits from the second screen, ('2').

13 shows an example of sharing or capturing a selected specific page 1304 or a specific image 1303 using only a hand-based touch on a screen 1301 or 1302 including a plurality of pages or images Giving. Specifically, when a pinch-in-touch gesture is performed while maintaining the hand-based touch input applied to the first point and the second point in a specific page or a specific image 1304 or 1303 selected, a sharing or capturing function The tool box 1310a or 1310b for popping up a pop-up window.

While holding the hand-based touch input applied to the first point and the second point, flicking another finger, for example, the index finger, which has not been touched, up and down to select a specific tool included in the tool box 1310a or 1310b, You can easily choose a sharing tool. Accordingly, there is an advantage that a desired operation can be easily performed even in an environment where it is difficult to use both hands and the pen device 200 freely.

As described above, according to the present invention, it is possible to easily call a desired input tool by performing a hand-based touch and a tool-based touch in parallel, and more natural and intuitive handwriting input can be performed. In addition, it is possible to perform seamless and continuous handwriting input in parallel with the hand-based touch and the tool-based touch without repeatedly calling and returning each time the input tool is used. It also provides more intuitive interactivity by providing a UI like the use of real tools in parallel with hand-based touches and tool-based touches.

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). Also, the computer may include a control unit 180 of the terminal. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

Claims (15)

A touch screen capable of receiving a first input signal and a second input signal based on different input methods and outputting a first screen corresponding to a first application;
And a controller for calling the input tool corresponding to the first input signal when the first input signal is received while displaying the first screen and displaying the graphic object representing the input tool on the first screen and,
Wherein,
When the second input signal is received while the first input signal is maintained and the graphic object is displayed, based on the area where the graphic object is output, the second input signal is input to the second input signal using an input tool corresponding to the graphic object And controls the touch screen so that a corresponding display is output. The method according to claim 1,
The control unit
When the touch corresponding to the first input signal is released,
Wherein the control unit controls the touch screen such that the graphic object disappears from the first screen and an indication corresponding to a subsequent second input signal is output as a pre-call attribute value of the input tool. The method according to claim 1,
Wherein the type of the input tool called on the basis of the first input signal is determined differently according to the touch method corresponding to the first input signal and the type of the first screen. The method according to claim 1,
Wherein,
When the touch corresponding to the first input signal is dragged while the graphic object is displayed, the graphic object is moved along the dragged path,
And controls to display an indication corresponding to the second input signal using the graphic object at the changed position. The method according to claim 1,
Wherein,
Controls the touch screen so that a display corresponding to the second input signal is aligned with a boundary line of the graphic object when a touch corresponding to the second input signal is dragged along an edge of the graphic object,
And limits the display output corresponding to the two input signals while a touch corresponding to the second input signal is being dragged in the graphic object. The method according to claim 1,
If the graphics object is changed as the first input signal is maintained and a subsequent first input signal is received while the graphic object is being displayed,
And outputs a display corresponding to the second input signal using an attribute corresponding to the changed graphic object. The method according to claim 6,
Wherein,
Displaying the graphic object while a first touch corresponding to the first input signal is maintained,
And changes the input tool corresponding to the graphic object differently when a second touch corresponding to a subsequent first input signal is applied while maintaining the first touch. The method according to claim 6,
Wherein,
Wherein the magnification of the input tool corresponding to the first screen and the graphic object is adjusted based on the touch degree of the first touch corresponding to the first input signal. The method according to claim 6,
Wherein,
Wherein the scale of the input tool corresponding to the graphic object is adjusted differently based on a direction and degree of a plurality of taps corresponding to the first input signal being dragged. The method according to claim 1,
Wherein the first screen comprises at least one object,
Wherein,
And a second graphic object for holding a display of the object and indicating an input tool applicable to the object when the second input signal is received in a state where a touch corresponding to the first input signal is applied to the object, And outputs it to the screen. 11. The method of claim 10,
Wherein,
When the second graphic object is output and a subsequent second input signal is received while the first input signal is maintained,
Mapping an attribute of the input tool corresponding to the second graphic object to a touch target object corresponding to the subsequent second input signal and mapping the attribute of the input tool corresponding to the subsequent second input signal to the object And controlling the mobile terminal. 12. The method of claim 11,
Wherein a result of using the input tool corresponding to the subsequent second input signal is not displayed on another object of the first screen while the first input signal is maintained. 12. The method of claim 11,
Wherein,
Releases the attribute of the input tool mapped to the touch object when the touch corresponding to the first input signal is released,
Wherein when a touch corresponding to the first input signal is applied to another object of the first screen, a third graphic object indicating the input tool applicable to another object is output to the first screen. The method according to claim 1,
Wherein the first screen includes a handwriting input using at least one of the first and second input signals,
Wherein,
When the first input signal is held and a drag touch input corresponding to the second input signal is applied to the area where the handwriting input is output while the graphic object is displayed in the area where the handwriting input is output, And arranges the handwriting input so as to correspond to a drag trajectory of the handwriting input. The method according to claim 1,
Wherein the first screen comprises at least one object,
Wherein,
When the object is selected using the first touch corresponding to the first input signal, outputs a first icon indicating an input tool related to the selected object,
When a touch is applied to the first icon by using a second touch corresponding to the first input signal while maintaining the first touch, an execution result of a function corresponding to the first icon in one area of the first screen Respectively,
Wherein when the second input signal is received, the execution result of the function is applied to the touch point at which the second input signal is received.

Images