KR20140092459A - Method for exchanging data between memo layer and application and electronic apparatus having the same - Google Patents

Abstract

The present invention provides a method and an electronic device for implementing natural conversation-type feedbacks. According to the present invention, in a terminal supporting a memo function using a touch panel, an application supporting the memo function is linked with another application so that written text images of the application supporting the memo function and feedbacks of the other application can be exchanged.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for exchanging data between a memo layer and an application,

The present invention relates to a terminal including a touch function, and more particularly, to a terminal including a touch function, a memo layer supporting a memo function using a touch panel is associated with an application already executed under the memo layer, And more particularly, to a method and an apparatus for exchanging data with a mobile terminal.

2. Description of the Related Art [0002] With the recent increase in portable electronic devices, a demand for a user interface (UI) capable of intuitive input / output is increasing. For example, in the conventional UI method of inputting information using a separate constitution such as a keyboard, a keypad, and a mouse, the UI is gradually evolved into an intuitive UI method of directly touching the screen using a finger or an electronic pen or inputting information using voice .

In addition, UI technology is not only convenient for users but also intuitive and human-centered. The representative UI technology enables a user to talk with a portable electronic device by voice, thereby inputting desired information or obtaining desired information.

In general, a smartphone, which is a portable electronic device of a typical user, can install various applications and use new functions through the installed applications.

However, since an application installed in a smart phone is generally operated by itself, it is not common that a plurality of applications are linked to provide a new function or a result to a user.

For example, even in a terminal supporting an intuitive UI in an application supporting schedule management, information can be input using only the UI supported by the application.

In addition, a method of using a memo content in connection with other applications is provided in a terminal that supports a memo function by using a touch panel, but is mainly utilized only for the purpose of memoing information by means of input means such as a finger or an electronic pen It is not.

In addition, the terminal is provided with an input means using an electronic pen such as a stylus pen in addition to the touch of the user's finger or the like. The user can use a more detailed touch or the like by using the electronic pen input means. However, there are few cases in which the electronic pen input means is used except for special cases because the functions that can be handled by the finger are well supported in operating the terminal.

Further, since the pen input means remains at a level of simply touching a specific area displayed on the display unit, various functions of the terminal are required to be operated based on the pen input means.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a method and apparatus in which an application that supports a memo function in a terminal supporting a memo function using a touch panel is linked to another application and receives feedback from the other application.

A method of controlling an application of an electronic device including a touch screen according to an exemplary embodiment of the present invention includes displaying an application on the touch screen, displaying a predetermined gesture on the touch screen during execution of the application, A memo layer including a handwriting input area for displaying data to be provided to the application and a display area for displaying data provided from the application is displayed superimposed on the application, An input image is displayed on the handwriting input area, the input handwriting image is recognized and compared with preset data, and feedback from the application is displayed in the display area of the memo layer according to the comparison result.

According to one aspect, the handwritten image and the feedback may be composed of a plurality of, and the feedback corresponding to the handwritten images appearing in the handwriting input area may be displayed in different display areas, respectively.

According to one aspect, when the memo layer is formed in a semi-transparent state and overlapped on the screen in which the application is running, the application may be displayed through the memo layer.

According to one aspect, the memo layer may be composed of a plurality of memo layers, and the plurality of memo layers may overlap and appear in multiple layers according to feedback received from the application.

According to one aspect, the feedback may be composed of a plurality, and may appear in a part of the memo layer.

According to one aspect, the plurality of feedbacks may be formed by an instruction that executes the application, and when a partial region in which the plurality of feedbacks is displayed is selected, the plurality of instructions in the selected partial region may be executed.

According to one aspect, when another application is executed instead of the application, another memo layer linked with the other application can be executed.

According to one aspect, the feedback displayed on the other memo layer may be displayed differently depending on the function of the other application.

According to one aspect, the handwritten image and the feedback may appear sequentially in some areas of the memo layer.

According to one aspect, the handwritten image and the feedback may be interactive.

According to one aspect, the handwritten image or feedback may be at least one of a character, a feedback, an image, a pattern, a number, a special character, or an email address.

An electronic device including a touch screen according to another embodiment of the present invention includes a handwriting input area for displaying an application to be displayed on the touch screen, A memo layer including a display area in which data provided from the application is displayed; a handwritten image is input to the handwriting input area of the memo layer and displayed in the handwriting input area; Detecting a predetermined gesture on the screen, comparing the input handwritten image with recognized and pre-set data, and providing feedback from the application to the display area of the memo layer according to the comparison result.

According to one aspect, the handwritten image and the feedback may be composed of a plurality of, and the feedback corresponding to the handwritten images appearing in the handwriting input area may be displayed in different display areas, respectively.

According to one aspect, the feedback may be composed of a plurality, and may appear in a part of the memo layer.

According to one aspect, the plurality of feedbacks may be formed by an instruction that executes the application, and when a partial region in which the plurality of feedbacks is displayed is selected, the plurality of instructions in the selected partial region may be executed.

According to one aspect, when another application is executed instead of the application, another memo layer linked with the other application can be executed.

According to one aspect, the feedback displayed on the other memo layer may be displayed differently depending on the function of the other application.

In a terminal supporting a memo function using a touch panel, a memo function application and another application are linked to each other to receive a handwritten image of the memo function application and feedback of the other application, / RTI >

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic view showing the configuration of an electronic device according to an embodiment of the present invention; Fig.
2 is a detailed diagram illustrating the configuration of a terminal 100 for supporting application control of an electronic device according to an embodiment of the present invention.
3 is a detailed block diagram of the controller 160 of the present invention.
FIG. 4 illustrates an exemplary configuration of a command processing unit 167 for supporting handwritten NLI technology in a terminal according to an embodiment of the present invention.
5 is a diagram illustrating an application and a memo layer being controlled using a driving engine according to an embodiment of the present invention.
6 is a flowchart for explaining a correlation between an application and a memo layer according to an embodiment of the present invention.
7 is a diagram for explaining a dialog between a sub-application and a memo layer according to an embodiment of the present invention.
8 is a diagram for explaining a dialog between a sub-application and a memo layer according to another embodiment of the present invention.
9 is a view for explaining a method of sending a message to a plurality of users using a sub-application and a memo layer according to another embodiment of the present invention.
10 is a diagram for explaining a method of changing information of a sender using a memo layer as an application example of Fig.
11 is a diagram for explaining a method of calling an emoticon image using a memo layer.
12 is a diagram for explaining a method of giving feedback to an item in which the memo layer is required when a part of the mail is empty when executing mail sending.
13 is a diagram showing a method of registering a web page bookmark using a memo layer.
14 is a diagram illustrating a method of sharing a web page using a memo layer with another user.
15 is a diagram illustrating a method of sorting files in a My Folder using a memo layer.
16 is a diagram illustrating a method of arranging images in a gallery in which picture files are stored using a memo layer.
17 is a diagram showing a method of setting an alarm using a memo layer.
18 is a diagram illustrating a method of registering an account using a memo layer.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals or signs refer to the same components or parts, and the sizes and thicknesses of components in the drawings may be exaggerated for convenience of illustration.

In the following description of the embodiments of the present invention, descriptions of techniques which are well known in the technical field of the present invention and are not directly related to the present invention will be omitted. In addition, detailed description of components having substantially the same configuration and function will be omitted.

For the same reason, some of the elements in the accompanying drawings are exaggerated, omitted, or schematically shown, and the size of each element does not entirely reflect the actual size. Accordingly, the present invention is not limited by the relative size or spacing depicted in the accompanying drawings.

In the embodiment of the present invention described below, a handwriting-based UI technique (hereinafter referred to as "handwriting-based NLI technique") is applied to a terminal as a natural language interface (NLI) Function to enable the user to perform the dialogue procedure with the user.

The NLI technology usually includes comprehension and generation capabilities, which allow the computer to understand what is entered and display text for human comprehension.

Thus, it is an application of natural language understanding that enables conversation between human and electronic devices in natural language.

For example, the terminal performs a command input by the user using the handwritten NLI technology or acquires information necessary for performing the input command from the user through a dialogue procedure.

As described above, in order to apply the handwriting-based NLI technology to the terminal as described above, it is desirable that the switching between the memo mode and the command processing mode be organically performed through the handwritten NLI technology. The memo mode includes a state of supporting a memo using an input means such as a finger or an electronic pen on a screen supported by an application running on the terminal. In addition, the command processing mode supports an operation of processing the memo contents inputted through the memo mode in association with the information according to the currently running application.

Meanwhile, as an example of a method of switching between the memo mode and the command processing mode, it is possible to switch a mode by pressing a button provided on the electronic pen, that is, by generating a hardware signal.

In the following description, the electronic pen will be described as typical input means for supporting the memo function. However, the present invention according to the following description is not applicable to a terminal using an electronic pen as an input means.

In other words, it is apparent that all of the embodiments proposed in the present invention can be used as input means by any means capable of inputting information on the touch panel.

In the present invention, it is preferable that the terminal share information with each other in advance in order to receive the desired information through the dialog with the user using the handwritten NLI technology and to provide the result of the processing to the user.

For example, it may be promised to use at least one of a symbol, a pattern, a character, a combination thereof to request switching of the operation mode, or it may be promised to use one operation through the operation recognition function. The switching of the typical operation mode to be requested at this time may be a transition from the command processing mode in the memo mode or the memo mode in the command processing mode.

On the other hand, it may be desirable to analyze the memo pattern of the user when promising the input information corresponding to the symbols, patterns, characters, combinations thereof, etc., and to determine the result based on the analyzed result. This will provide the user with the convenience of intuitively entering the desired information.

In the following detailed description of the present invention, scenarios for controlling the running application through a memo function by applying handwritten NLI technology in a situation where various applications are driven and outputting the results are described in detail as different embodiments will be.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic view showing the configuration of an electronic device according to an embodiment of the present invention; Fig.

Referring to FIG. 1, an electronic device 10 of the present invention may include a terminal 100 and an electronic pen 200.

The electronic device 10 of the present invention having the above-described structure is configured such that when performing a specific input operation on the display unit 140 of the terminal 100 while holding the electronic pen 200, And a gesture recognition of the electronic pen 200. The user function corresponding to the specific pen function command can be performed based on the gesture recognition of the electronic pen 200. [

In addition, the electronic device 10 of the present invention may be configured to determine whether or not a pen corresponding to a specific pen function command is selected based on the pen state of the electronic pen 200 and the gesture recognition value of the electronic pen 200, User functions can be supported to be performed.

Accordingly, the electronic device 10 of the present invention can support a user to perform various input operations of the terminal more easily and quickly using the electronic pen 200. [

The terminal 100 may include a display panel (not shown) for outputting a screen according to the function operation and a pen recognition panel (not shown) for operating the electronic pen 200.

In addition, the terminal 100 may further include a touch panel for supporting a touch event, and includes a drive module for driving the display panel and the pen recognition panel, a power source, and a storage unit for mounting various elements constituting the terminal .

The terminal 100 may collect gesture information by recognizing a gesture corresponding to a pen touch event collected from the pen recognition panel and collect pen state information transmitted from the electronic pen 200. [ The terminal 100 supports the user function operation corresponding to the specific pen function command predefined based on the collected gesture information and the pen status information. At this time, the terminal 100 may support a user function operation corresponding to another predefined command according to at least one application program type currently activated.

The detailed configuration and functioning of the terminal 100 will be described in more detail with reference to the following drawings.

The electronic pen 200 may be composed of a pen base 220 and a pen tip 210 disposed at the end of the pen base 220. The electronic pen 200 further includes a coil 230 disposed inside the pen 220 of the area adjacent to the pen tip 210 and a button 240 for changing the electromagnetic induction value generated by the coil 230 And the like. The electronic pen 200 according to the present invention having such a configuration supports the electrostatic induction method, and a magnetic field is formed at a certain point of the pen recognition panel 143 by the coil 230, It is possible to detect the magnetic field position and to recognize the touch point.

Here, the pen point 210 is configured to contact the touch panel when the touch panel is disposed on the display panel or the display panel, and to indicate that the electronic pen 200 indicates a specific point on the display unit 140 . ≪ / RTI >

Since the pen tip 210 is disposed at the end of the pen base 220 and the coil 230 is disposed at a position spaced apart from the pen point 210 by a predetermined distance, an input operation such as writing is performed while gripping the electronic pen 200 The distance compensation between the contact point of the pen tip 210 and the magnetic field forming point by the coil 230 can be performed.

The user can perform an input operation such as writing or drawing operation, item selection and placement, etc., while pointing the pen tip 210 at a certain point on the display panel 141 according to the distance compensation described above. Particularly, the user can perform a specific gesture input while the pen tip 210 is in contact with the display panel.

When the electronic pen 200 approaches the pen recognition panel within a predetermined distance, a magnetic field generated by the coil 230 can be generated at a predetermined position of the pen recognition panel.

To this end, the terminal 100 may perform a magnetic field scan operation on the pen recognition panel in real time or at regular intervals. Also, the activation of the pen recognition panel may be performed at the time of activation of the electronic pen 200 operation mode.

In particular, the pen recognition panel may recognize the pen state differently depending on a proximity distance at which the electronic pen 200 approaches the pen recognition panel. That is, when the pen recognition panel is positioned within a first predetermined distance from the display panel or the pen recognition panel, the electronic pen 200 can be recognized as a contact state.

In addition, the pen recognition panel may recognize the electronic pen 200 as a hovering state when the electronic pen 200 is disposed within a range of the first predetermined distance or more and a second predetermined distance or less. In addition, when the electronic pen 200 is positioned within a recognizable range within a range of the second predetermined distance or more from the pen recognition panel, it can be recognized as an air state. As described above, the pen recognition panel of the terminal 100 can provide various pen status information according to the distance from the electronic pen 200. [

Meanwhile, the button 240 provided on the electronic pen 200 can be pressed by the user. As the button 240 is pressed, a specific signal value is generated in the electronic pen 200 and transmitted to the pen recognition panel .

To this end, a specific device capable of changing a specific capacitor, additional coil or electrostatic induction may be disposed in an adjacent area of the button 240, and the corresponding device may be mounted on the coil 230 To change the electromagnetic induction value induced in the pen recognition panel to recognize the depression of the button 240 of the electronic pen 200. [

Or the button 240 may be generated and the generated radio signal may be transmitted to a receiver provided in a separate area of the terminal 100. The terminal 100 may be designed to transmit the radio signal It is possible to recognize that the button 240 of the pen 200 is pressed.

As described above, the terminal 100 can collect various pen state information according to the arrangement state of the electronic pen 200 described above. That is, the terminal 100 receives information on whether the electronic pen 200 is in a contact state or a hovering state, information on whether the button 240 of the electronic pen 200 is pressed or maintained, .

The terminal 100 generates a specific pen function command based on the pen state information provided by the electronic pen 200 and the gesture recognition information corresponding to the gesture input operation input by the coil 230 of the electronic pen 200 , It can support function execution corresponding to the command.

As described above, the electronic device 10 of the present invention can include the terminal 100 and the electronic pen 200. The electronic device 100 may process pen status information including the arrangement state of the electronic pen 200 and a specific input state of the electronic pen 200, and various gesture recognition information according to the gesture input operation.

The electronic device 100 can process instructions for various functions provided by the terminal 100 or the electronic pen 200 so that the commands can be generated and operated more easily and quickly.

In addition, the electronic device 10 of the present invention can support the specific function of the terminal 100 more quickly in a specific pen state based on the compound gesture recognition information. Alternatively, the electronic device 10 of the present invention may perform compound gesture recognition independently of a constant pen state and support the performance of the function.

2 is a detailed diagram illustrating the configuration of a terminal 100 for supporting application control of an electronic device according to an embodiment of the present invention.

2, the terminal 100 of the present invention includes a communication unit 110, an input unit 120, an audio processing unit 130, a display panel 141, a pen recognition panel 143, a touch panel 145, Unit 150 and a controller 160. The controller 150 may be a microprocessor. Here, the terminal 100 of the present invention includes the touch panel 145, but the present invention is not limited thereto.

That is, the terminal 100 of the present invention may be provided in a form in which the touch panel 145 is removed as described above with reference to FIG. In this case, the terminal 100 generates an input signal for operating the terminal 100 using only the electronic pen 200 and the pen recognition panel 143, or inputs the input signal based on the input unit 120, the communication unit 110, Lt; / RTI > signal.

The terminal 100 of the present invention having the above-described configuration can collect the pen status information of the electronic pen 200 and the gesture recognition information corresponding to the gesture input operation based on the pen recognition panel 143. The terminal 100 can confirm predetermined pen function commands corresponding to the collected pen state information and gesture recognition information, and perform terminal function support according to the pen function commands.

In addition, the terminal 100 collects information on the function type of the currently active application program as well as the pen status information and the gesture recognition information, and stores the pen status information, the gesture recognition information, Or to generate functional instructions.

Alternatively, the terminal 100 may support performing a specific terminal function according to the compound gesture recognition information when the gesture recognition information is compound gesture recognition information.

To this end, the pen recognition panel 143 may be disposed at a predetermined position of the terminal 100 and may have an active state according to a specific event or by default. The pen recognition panel 143 may be provided under the display panel 141 with a predetermined area, for example, an area capable of covering the display area of the display panel 141. The pen recognition panel 143 may receive the pen status information according to the approach of the electronic pen 200 or the operation of the electronic pen 200 and may transmit the pen status information to the control unit 160. In addition, the pen recognition panel 143 may receive the gesture recognition information according to the gesture operation of the electronic pen 200 and may transmit the gesture recognition information to the controller 160.

The pen recognition panel 143 may be provided to receive the position value of the electronic pen 200 according to the electromagnetic induction method with the electronic pen 200 having the coil 230 as described above. The pen recognition panel 143 may collect the electromagnetic induction values corresponding to the intervals according to the access intervals of the electronic pen 200, and may transmit the electromagnetic induction values to the controller 160. The electromagnetic induction value transmitted at this time indicates whether the electronic pen 200 is in a hovering state in which the electronic pen 200 is spaced apart from the pen recognition panel 143 or the display panel 141 or the touch panel 145, 141 or the touch panel 145. The touch panel 145 may be in contact with the touch panel 145 at a predetermined interval.

Meanwhile, the pen state information collected by the terminal 100 according to the design of the button 240 provided in the electronic pen 200 may be implemented differently. That is, if the button 240 is provided with a structure for changing the electromagnetic induction value formed by the coil 230 as described above, the pen state information about whether or not the button 240 is input is determined by the pen recognition panel 143 And can transmit the information to the control unit 160. For example, it may be a capacitor selectively connected to the button 240 as a structure for changing the electromagnetic induction value, a separate coil, or the like, and may be used for various types of electromagnetic induction Certain devices that can change the value can be applied.

If the button 240 is provided with a separate wireless signal transmission structure, the terminal 100 may further include a receiving device capable of receiving a radio signal corresponding to the input of the button 240, It is possible to determine whether or not the button 240 is input based on the radio signal received by the receiving apparatus.

The touch panel 145 may be disposed at an upper portion or a lower portion of the display panel 141. The touch panel 145 may include information about a touch position and a touch state according to a change in capacitance due to a touch of a touch object, 160).

The touch panel 145 may be provided in at least a part of the display panel 141 and may be deactivated at the time when the pen recognition panel 143 is activated according to the designing method. That is, the touch panel 145 according to the present invention can generate a command generation function based on the pen recognition panel 143 such as the pen status information and the gesture recognition information, the pen status information, the gesture recognition information, the function type information, You can have an inactive state while supporting it.

The display panel 141 is configured to output various screens related to the operation of the terminal 100. For example, the display panel 141 displays an initial idle screen or a menu screen for supporting the function of the terminal 100, a file search screen, a file playback screen, a broadcast reception screen, a file edit screen, Various screens such as a screen, a memo creation screen, an e-book reading screen, a chat screen, an e-mail or a message creation and reception screen can be provided according to the function activation.

Each of the screens provided by the display panel 141 may have information on a specific function type, and the function type information may be transmitted to the controller 160. [

When each function of the display panel 141 is activated, the pen recognition panel 143 can be activated according to a preset setting. On the other hand, the gesture recognition information input through the pen recognition panel 143 may be output to the display panel 141 in a form corresponding thereto. For example, when the gesture recognition information is a gesture corresponding to a specific pattern, an image corresponding to the pattern may be output to the display panel 141. [

The user can check the gesture and check which gesture he or she is entering.

In particular, the gesture input of the present invention can confirm the input start time and the input completion time based on the change of the pen state information of the electronic pen 200. That is, the gesture input may start at least one of the contact state or the hovering state of the electronic pen 200, and the input completion time may be a state where either the contact state or the hovering state is released.

Accordingly, the user can perform the gesture input operation on the display panel 141 using the electronic pen 200 or while being spaced apart from the display panel 141 by a predetermined distance. For example, when the electronic pen 200 moves in a range of a contact state, the terminal 100 recognizes the input as a gesture input and performs gesture recognition according to the movement.

When the electronic pen 200 is within the range where the electronic pen 200 is recognized as hovering, the terminal 100 does not recognize the gesture input according to the motion, but can recognize the valid motion as the gesture input continuously.

Accordingly, the user touches and moves the display panel 141 with the electronic pen 200 to perform an operation for inputting the actual gesture, separates the electronic pen 200 from the display panel 141, You can perform complex gesture input in a moving manner.

The storage unit 150 is a configuration capable of storing various programs and data necessary for the operation of the terminal 100 of the present invention. For example, the storage unit 150 may store an operating system necessary for the operation of the terminal 100, and may store functional programs for supporting a screen displayed on the display panel 141 described above.

In particular, the storage unit 150 may store the pen function operation program 151 for supporting the pen function operation of the present invention, and may store the pen function table 153 for supporting the pen function operation program 151. [

The pen function operating program 151 may include various routines for supporting the pen function operation of the present invention. For example, the pen function operating program 151 may include a routine for confirming an activation condition of the pen recognition panel 143, a routine for collecting pen status information of the electronic pen 200 upon activation of the pen recognition panel 143, 200), and collect gesture recognition information.

The pen function operating program 151 may include a routine for generating a specific pen function command based on the collected pen state information and the gesture recognition information, and a routine for performing a function according to the pen function command.

The pen function operating program 151 further includes a routine for collecting the type information of the currently activated function and generates a pen function command that is mapped to the acquired type information of the function, pen state information, and gesture recognition information Routine, and a routine for performing a function according to the corresponding pen function command.

In addition, the pen function operating program 151 may include a routine for recognizing the input gesture input operation as an input operation for the compound gesture recognition information and separately recognizing the input operation as separate sub gesture recognition information, a divided sub gesture recognition A routine for generating a specific pen function command based on the information, and a routine for performing a function according to the generated pen function command.

In the above description, the routine for generating the pen function command is a configuration capable of generating an instruction with reference to the pen function table 153 stored in the storage unit 150. [ The pen function table 153 may be a table including a pen function instruction to which a specific terminal function operation corresponding to an input operation by the electronic pen 200 is mapped by a designer or by a program developer.

Particularly, the pen function table 153 maps the gesture recognition information input according to the pen status information and the function type information to each pen function command, so that even if the gesture recognition information is the same gesture recognition information, . ≪ / RTI >

In addition, the pen function table 153 may include information in which pen function commands corresponding to a specific terminal function operation are defined in the pen state information and the gesture recognition information.

The pen function table 153 including only the pen state information and the gesture recognition information can support the specific function operation based only on the pen state information and the gesture recognition information regardless of the type of the currently activated function. Meanwhile, the pen function table 153 may include compound gesture recognition information and pen function instructions that are mapped to compound gesture recognition information.

The pen function table 153 configured by the compound gesture recognition information can support the specific terminal function support based only on the input compound gesture recognition information regardless of the pen state information or the type of the function currently provided.

As described above, the pen function table 153 includes a first pen function table including pen state information, function type information, and pen function commands mapped to gesture recognition information, a pen state information and a pen mapped to the gesture recognition information A second pen function table including function commands, and a third pen function table including pen function commands mapped to compound gesture recognition information.

The pen function table 153 including the respective pen function commands can be selectively or automatically applied according to the user setting or the type of the application program being executed. For example, the user can set whether to use the first pen function table or the second pen function table.

Then, the terminal 100 can support the gesture recognition process corresponding to the input operation based on the specific pen function table according to the setting.

In addition, the terminal 100 can support applying the second pen function table according to a design method or when a first application program is activated according to a user setting, and when the second application program is activated, 3 pen function table, and may support applying the first pen function table when the third application program is activated. As described above, the pen function table 153 of the present invention can be variously applied according to the type of function being activated.

Meanwhile, the communication unit 110 may be included when the terminal 100 supports a communication function. In particular, when the terminal 100 supports the mobile communication function, the communication unit 110 may be configured as a mobile communication module.

The communication unit 110 may perform a chat function, a message transmission / reception function, a call function, and the like, for example, as a terminal function for which a communication function is required. In the course of operation of the communication unit 110, When the information is collected, the control unit 160 may support execution of the pen function command corresponding to the gesture recognition information.

The communication unit 110 may receive the update information of the pen function table 153 of the present invention from an external device while supporting the communication function of the terminal 100 and may transmit the updated information to the control unit 160.

The pen function table 153 described above can be set differently according to the terminal function type by the executed application program. As a result, when the terminal 100 installs a new function, a new setting related to the operation of the electronic pen 200 may be required.

The communication unit 110 can support reception of information on the pen function table 153 by default or upon a user request when the pen function table 153 for a new function or a previously installed function is provided.

The input unit 120 may include a side key or a separate touch pad. Also, the input unit 120 may include a button key for performing turn-on or turn-off of the terminal 100, a home key for supporting return to the basic screen supported by the terminal 100, and the like.

In addition, the input unit 120 may generate an input signal for setting the pen function operation mode of the present invention according to the user's control, and may transmit the input signal to the controller 160.

That is, the input unit 120 may include a basic pen function operation mode for performing a function according to the position of the pen, a pen position, and a pen function table 153, which are based on any one of the above- An input signal for setting any one of the function operation modes can be generated.

The terminal 100 calls the specific pen function table 153 according to the input signal and can support the pen function operation based on the specific pen function table 153. [

The audio processing unit 130 includes at least one of a speaker SPK for outputting audio signals of the terminal 100 and a microphone MIC for collecting audio signals.

The audio processing unit 130 may output a guidance sound or an effect sound to guide the pen operation mode setting according to the present invention. When the gesture recognition information according to the specific gesture operation is collected in the pen recognition panel 143, the audio processing unit 130 supports the guidance sound corresponding to the gesture recognition information or the effect sound according to the performance of the function.

The audio processing unit 130 may support output of sound effects for a gesture input in real time during a gesture input operation. In addition, the audio processing unit 130 may control the vibration module to control the vibration amplitude adjustment of the vibration module corresponding to the gesture input operation.

At this time, the audio processor 130 may vary the magnitude of the vibration according to the input gesture input operation. That is, the audio processing unit 130 can control the gesture recognition information to be performed so that the gesture recognition information has a vibration magnitude corresponding to each gesture recognition information.

The audio processing unit 130 may perform the magnitude or type of the effect sound differently according to the type of the gesture recognition information even in the process of outputting the sound effect. For example, when the gesture recognition information related to the currently performed function is collected, the audio processing unit 130 supports vibration and sound output having a predetermined size, and when the gesture recognition information for calling another function is collected, Can support vibration or sound output of the sound.

The controller 160 may include various components for supporting a pen function operation or an interaction between a plurality of programs according to an embodiment of the present invention. Signal processing, data processing, and function operation can be controlled. For this, the control unit 160 may include a configuration as shown in FIG.

3 is a detailed block diagram of the controller 160 of the present invention.

3, the control unit 160 includes a function type checking unit 161, a pen status checking unit 163, a gesture recognizing unit 165, a command processing unit 167, and a function performing unit 169 Configuration.

The function type checking unit 161 confirms the type of the user function currently being activated in the terminal 100. [ In particular, the function type identifying unit 161 collects the type information of the function related to the screen currently being displayed on the display panel 141. [

That is, in the case of the multitasking function supporting terminal 100, a plurality of functions can be activated according to activation of a plurality of application programs. In this case, the function type determination unit 161 may collect only the type information of the function related to the screen currently displayed on the display panel 141 and provide the information to the command processing unit 167. [

In this process, if there are a plurality of screens output on the display panel 141, the function type identifying unit 161 can collect the type information of the function related to the screen output to the uppermost layer.

The pen status confirmation unit 163 is a configuration for collecting information corresponding to the position of the electronic pen 200, the depression of the button 240, and the like. As described above, the pen state confirmation unit 163 detects a change in the electromagnetic induction value input by scanning the pen recognition panel 143, and determines whether the electronic pen 200 is in a hovering state, a contact state, It is possible to confirm whether the pressed state is the pressed state or not, and perform pen state information collection corresponding thereto. The collected pen status information may be provided to the command processing unit 167. [

The gesture recognition unit 165 recognizes the gesture according to the movement of the electronic pen 200. [ The gesture recognizing unit 165 recognizes the gesture operation according to the movement of the electronic pen 200 regardless of the hovering state or the contact state of the electronic pen 200 and outputs the gesture recognition information to the command processing unit 167 ).

The gesture recognition information provided by the gesture recognition unit 165 may include single gesture recognition information recognized as one object and composite gesture recognition information recognized as a plurality of objects. The distinction can be determined by the definition of the gesture input operation process.

For example, an input action corresponding to a motion that is continuously occurring in a contact state maintaining state or a hovering state holding state can generate single gesture recognition information.

In addition, the input operation corresponding to the motion generated when the connection state and the hovering state are changed can generate composite gesture recognition information. Alternatively, when the hovering state is changed to the air state and the movement occurs, the gesture recognition unit 165 can recognize the composite gesture recognition information.

Alternatively, the gesture recognition unit 165 can determine that a plurality of gesture motions are generated based on a boundary where the electronic pen 200 is out of the recognizable range from the pen recognition panel 143, as an operation for recognizing the complex gesture have.

The gesture recognition unit 165 may output a button map for inputting a separate gesture so that the user can designate completion of the gesture input when the electronic pen 200 is recognized as a boundary based on the deviation from the pen recognition panel 143 .

The command processing unit 167 receives the function type information transmitted from the function type identifying unit 161, the pen state information transmitted from the pen state checking unit 163, the gesture recognition information transmitted from the gesture recognizing unit 165, The pen function command is generated by using at least one of the pen function commands.

In this process, the command processing unit 167 can refer to the pen function table 153 that defines various pen function commands.

In particular, the command processing unit 167 refers to the first pen function table corresponding to the function type information, the pen state information, and the gesture recognition information as described above according to the setting or the function type currently being activated, The second pen function table based on information and gesture recognition information may be referred to, or the third pen function table based on gesture recognition information may be referred to.

The pen function command generated by the command processing unit 167 may be transmitted to the function execution unit 169.

The function performing unit 169 controls the function operation corresponding to the pen function command generated and transmitted by the command processing unit 167. [ The function execution unit 169 can support a specific function operation related to a currently activated function, call a new function, or terminate a specific function or the like.

FIG. 4 illustrates an exemplary configuration of a command processing unit 167 for supporting handwritten NLI technology in a terminal according to an embodiment of the present invention.

Referring to FIG. 4, an instruction processing unit 167 supporting a handwriting-based NLI technique may include a recognition engine 410 and an NLI engine 420.

The cognitive engine 410 may include a Recognition Manager Module 412, a Remote Recognition Client Module 414 and a Local Recognition Module 416. The area recognition module 416 may include a handwriting recognition block, an optical character recognition block, and an operation recognition block.

The NLI engine 420 includes a dialog module 422 and an intelligence module 424.

The dialog module 422 is composed of a dialog management block for controlling the conversation flow and a natural language understanding block for grasping the intention of the user.

The intelligence module 424 includes a user modeling block for reflecting the user's preference, a common reasoning block for reflecting common sense, and a content management block for reflecting the user's situation.

The recognition engine 410 inputs information by a drawing engine (not shown) corresponding to input means such as the electronic pen 200 and an intelligent input platform (not shown) such as a camera can do.

The intelligent input platform may be an optical character recognizer such as an optical character reader (OCR) or the like, and may read information having a printed character or a letter, a number, or other symbol in the form of a hand written by a person, can do.

The drawing engine is a component for receiving input by various input means such as a finger, an object, and an electronic pen 200, and can input information by a handwriting image and an object.

Information input to the cognitive engine 410 may be input in lines, symbols, patterns, and combinations thereof in addition to characters.

The recognition engine 410 recognizes the memo contents of the area selected by the user among the memo contents currently displayed from the characters, lines, symbols, patterns, figures, combinations thereof, etc. inputted as information or recognizes the command instructed by the user .

The recognition engine 410 can output the recognition result obtained by the above-described method.

For this, the recognition engine 410 includes a cognition management module 412, a remote recognition request module 414, and a local recognition module 416 for recognizing input information, which processes overall control for outputting recognition results from the input information . The area recognition module 416 includes a handwriting recognition block for recognizing information input based on the handwriting, a light character recognition block for recognizing information from the input optical signal, and an operation recognition block for recognizing information from the input operation, .

The handwriting recognition block recognizes information inputted based on handwriting. For example, the handwriting recognition block recognizes the contents written by the electronic pen 200.

The optical character recognition block recognizes the optical character by receiving the optical signal sensed by the optical sensing module (not shown), and outputs the recognition result optical character recognition result value. The motion recognition block receives the motion detection signal detected by the motion detection module (not shown), recognizes the motion, and outputs the motion recognition result value.

The recognition result recognized by the handwriting recognition block, the optical character recognition block, and the motion recognition block may be provided to the NLI engine 420.

The NLI engine 420 can grasp the intention of the user through processing such as analysis of the recognition result provided from the recognition engine 410. [ That is, the NLI engine 420 can grasp the information that the user desires to input from the recognition result provided from the recognition engine 410.

The NLI engine 420 can collect the sufficient information by performing dialogue dialogue with the user using the handwritten NLI technology, and can grasp the intention of the user based on the collected information.

To this end, the dialog module 422 constituting the NLI engine 420 constructs a question for a conversation with a user, transmits the question to the user, and provides a flow of a dialog for receiving an answer or feedback from the user Can be controlled.

The dialog module 422 can manage the information obtained through the dialog (dialog management block). In addition, the conversation module 422 understands the intention of the user through the natural language processing of the command that is initially provided in consideration of the managed information (natural language understanding block).

The intelligent module 424 constituting the NLI engine 420 may generate information to be referred to in order to grasp the intention of the user through natural language processing and may provide it to the dialog module 422. [ For example, the intelligent module 424 may analyze the habits of the user in the memo to memorize information (user modeling block) reflecting the user's preference or infer information (common sense inference block) (Context management block).

Accordingly, the dialog module 422 constituting the NLI engine 420 can control the dialogue flow according to the dialog procedure with the user, with the help of the information provided from the intelligence module 424.

5 is a diagram illustrating an application and a memo layer being controlled using a driving engine according to an embodiment of the present invention.

For example, the configuration shown in FIG. 5 overlaps and displays a memo layer on the screen of a specific application running on the terminal 100, grasps the intention of the user according to the content written on the memo layer displayed, It is possible to control the operation of a specific application being executed.

The memo layer indicates an application capable of memo function using the touch panel.

At this time, the specific application being executed in the terminal is referred to as a 'sub-application'.

When the application supporting the memo function is executed during execution of the sub-application, the terminal 100 overlaps and displays the memo layer on the screen according to the execution of the sub-application. Referring to FIG. 5, the terminal 100 may include an application driving engine 510 and a memo layer driving engine 520.

The application driving engine 510 may execute an application requested by a user and perform an overall operation of controlling an application being executed by recognizing the intention of the user.

In particular, after the memo layer is called by the user, the application driving engine 510, when an operation command according to the contents written by the user is provided through the memo layer being called, Can be controlled.

To this end, the application driving engine 510 receives predetermined information from the memo layer driving engine (for example, a memory card) to guide the information required to control the operation of the running application at the present time 520).

The predetermined information may include at least one of a type of the application and a function menu currently being executed based on the application.

In this case, the application driving engine 510 may be provided with more accurate information on the application being executed in order to control the operation of the application currently being executed.

The memo layer driving engine 520 may continuously monitor whether or not the input according to the predetermined format is performed by the user in order to request execution of the application supporting the memo function.

The previously promised format may represent a touch event. For example, the previously promised format may be a user's touch dragging the screen on the screen on which the application is running. The touch drag means an operation of moving a finger or a touch tool to another position in the screen while the user holds the touch after touching the finger or the touch tool on the screen.

At this time, the touch drag may be performed from the left side to the right side of the screen or vice versa, or may be from the lower side to the upper side of the screen or vice versa.

Any tool capable of recognizing the touch drag can be used. Typically, it will be a user's finger or electronic pen 200 or the like.

When the memo layer driving engine 520 requests execution of an application for a memo function by a user, the memo layer driving engine 520 may call a memo layer capable of memo by the user. The memo layer driving engine 520 can display the called memo layer by overlapping on the screen according to the execution of the application.

At this time, it is preferable that the memorizer has transparency such that the execution screen of the application can be seen even when the application is overlapped on the execution screen of the application. At this time, transparency is an extent to which transparency is transparent from the transparent state to the opaque state, and is adjustable.

The area where the memo layer is overlapped on the screen can be set according to the request of the user. That is, the memo layer may be overlaid on the entire screen or may be overlapped on only some screens depending on the setting. In addition, after the memo layer is displayed so as to overlap only some screens, the user can change the size by selecting and dragging the outline or vertex of the memo layer.

As described above, the memo layer driving engine 520 can control an overall operation for displaying a memo layer on the screen, which can be memoed by a user's request while the application is being executed.

If the information providing guidance for the information required for controlling the application is provided from the application driving engine 510, the memo layer driving engine 520 displays 'input by the user' in the memo layer displayed on the screen A message that guides the requested information " can be displayed together.

For example, if the current music playback application is running, you can display the message 'Please write a song to play' in the note layer or 'Please write artist to search'.

The memo layer driving engine 520 may display the memo layer on the screen and recognize the intention of the user based on the content written by the user. At this time, the contents written by the user may be predetermined writing contents, and the memo layer driving engine 520 may recognize the writing contents and recognize the intention of the user. The memo layer driving engine 520 may provide relevant control information to the application driving engine 510 so that the application can be driven by the perceived user's intention.

The memo layer driving engine 520 may continuously provide relevant information to the application driving engine 510 to grasp the intention of the user.

The memo layer driving engine 520 may additionally display an input menu button on the menu layer displayed on the screen so that the user can recognize that the content input by handwriting is completed.

In this case, when the input menu button is pressed by the user, the memo layer driving engine 520 may start an operation for recognizing the intention of the user based on the contents written on the memo layer.

6 is a flowchart for explaining a correlation between an application and a memo layer according to an embodiment of the present invention.

In step 610, the control unit 160 may execute a specific application, that is, a sub-application, at the request of the user. After the sub-application is executed, the controller 160 may control the overall operation of the sub-application being executed. The application driving engine 510 that controls the sub-application may display a state of the operation of the sub-application on the screen of the terminal 100 so that the user can check the operation state of the sub-application.

In step 620, the control unit 160 may continuously monitor whether the memo layer is called by the user while the sub-application is being executed.

For example, the terminal 100 can call the memo layer by touching and dragging the screen on the screen on which the sub-application is running.

At this time, the touch drag may be performed from the left side to the right side of the screen or vice versa, or may be from the lower side to the upper side of the screen or vice versa. Any tool capable of recognizing the touch drag can be used.

Between steps 610 and 620, the memo layer receives the information from the lower application, recognizes the type of the character input to the memo layer, and displays the pattern in the upper area of the memo layer. The data displayed in the upper area of the memo layer may appear differently depending on the type or function of the lower application.

The user can recognize the type of contents input to the memo layer through data appearing in the upper area of the memo layer.

For example, if the sub-application is a program related to a song, the name displayed at the top of the memo layer may indicate that the song title should be written.

Also, if the sub-application is a program related to mail, the name displayed at the top of the memo layer may indicate that the name of the sender of the mail should be written.

The name may be displayed differently depending on the setting or the type of the sub-application. Also, it may be displayed constantly regardless of the type of the sub-application.

In step 630, the memo layer engine 520 that controls the memo layer continuously monitors whether the memo image is input to the memo layer.

In step 640, when a handwritten image is inputted through the input unit 120, the memo layer recognizes the handwritten image input and can pass the inputted handwritten image information to the lower application.

In step 650, the lower application driving engine 510 may compare the handwritten image information received through the lower application with the preset data stored in the storage unit 150. [

The handwritten image sent by the memo layer to the sub-application may be a feedback that may appear as if the user naturally spoke.

The preset data may be configured so that the feedback received and received between the memo layer and the sub-application becomes natural feedback. The pre-set data and the input handwritten image may be compared with each other so as to be natural feedback through the dialog module of the NLI engine 420, It is possible to search for a feedback having high relevance to the input handwritten image and transmit the feedback to the memo layer.

In step 660, if the feedback corresponding to the input handwritten image is found from the preset data in the sub-application, the corresponding feedback may be transmitted to the memo layer through the sub-application.

If the corresponding feedback can not be found from the already stored data, feedback may appear to re-enter the handwritten image since there is no corresponding feedback through the display 140 of the terminal 100.

Also, the control unit 160 may recognize that the handwriting image inputted by the user has an error, and send an error message to the memo layer with the feedback.

In addition, since another handwritten image can be input to the memo layer, the controller 160 can switch the memo layer to the stand-by mode 665 waiting for another handwritten image to be input to the input unit 120 again.

The handwriting image waiting state 665 may indicate a pause state until the other handwriting image is input through the input unit 120. [

In operation 670, when the corresponding feedback is received, the memo layer driving engine 520 may display the received feedback on the memo layer. At this time, the memo layer may be composed of a plurality of memo layers, and a plurality of memo layers may be superimposed and displayed in multiple layers according to the feedback received from the application.

The control unit 160 may repeat the steps 610 through 670 until a specific signal is received.

The particular signal may be a signal where the memo layer is terminated or deactivated or the corresponding handwriting image is executed.

In addition, the specific signal may be a signal in which another memo layer is executed or another sub-application is executed.

Accordingly, the specific signal may represent a signal other than the character or gesture used in step 610 in step 610.

7 is a diagram for explaining a dialog between a sub-application and a memo layer according to an embodiment of the present invention.

When the contact related icon 712 is selected on the display screen 711 as shown in 710, the application 721 corresponding to the icon 711 is executed to display the contact application screen 720 as shown in 720, Can be displayed.

720, when the user's gesture is input on the application 721, the memo layer 731 may be overlaid on the application 721 as shown in the screen 730. FIG.

Here, the application 721 may appear below the memo layer 731 and may be called a lower application.

The control unit 160 may display the received handwritten image 732 on a predetermined area of the display unit 140 when the handwritten image 732 is input to the memo layer 731. [

The NLI engine 420 of the control unit 160 may display a question on the screen requesting the handwritten image 732 using the dialog procedure with the user. For example, the NLI engine 420 may display "Merge with google account" on the screen to prompt the user to input the handwritten image 732 in response.

When the user makes a note of "Yes" in response to the memo function, the NLI engine 420 can recognize the intention of the user to execute by the input of the user.

In the screen 720, a plurality of pieces of information displayed in the application 721 can be linked to other accounts and the information can be shared with each other.

First, the memo layer 731 can be selected by selecting any of the plurality of pieces of information shown in the application 721. [ Then, the selected arbitrary information may be an object to be associated with the other account.

In addition, if the memo layer 731 is executed without selecting the information displayed in the application 721, the plurality of pieces of information can be objects to be linked with other accounts.

If Merge is input to the mero layer 731, the memo layer 731 can send the Merge information to the application 721. [

The application 721 can compare the received information with the already set data to search for data that can become the feedback 733. [

The search feedback 733 may be displayed through the memo layer 731. [ The handwritten image 732 may be at least one of letters, feedback, images, patterns, numbers, special characters, email addresses.

In addition, when the character receiving function is designed to support pen function operation, a handwritten image 732 input using the pen can be performed. At this time, the terminal 100 can recognize the information corresponding to the gesture input operation input from the electronic pen 200 in a state in which the handwriting image 732 is displayed on the memo layer 731.

If the predetermined time elapses, the handwritten image 732 may be set to disappear. In the memo layer 731, when the handwritten image 732 disappears, the input function by the electronic pen 200 can be designed not to be supported. Accordingly, the terminal 100 can support the handwriting image display waiting time for the purpose of supporting the pen function operation or the like.

For this, the terminal 100 may provide a menu screen for adjusting the handwriting image display waiting time. The user can arbitrarily adjust the display waiting time of the handwritten image according to his or her preference using the display wait time adjustment menu screen so as to correspond to his or her reply writing ability. The terminal 100 may define the display waiting time of the handwritten image according to a predetermined method.

Using the control unit 160, the memo layer 731 can receive the handwritten image 732 obtained and transmit the handwritten image 732 to the sub-application 721. [

The sub application 721 compares the handwritten image 732 acquired using the NLI engine 22 and the sub application engine 510 with preset data and selects a feedback 733 suitable for the handwriting image 732 .

The feedback 733 may be at least one of a character, a feedback, an image, a pattern, a number, a special character, and an e-mail address.

When the feedback 733 is selected, the appropriate feedback 733 may be transmitted to the memo layer 731 via the sub-application 721 so that it can be displayed on the memo layer 731. [

The feedback signal 733 is displayed on the memo layer 731 and an audio signal may be output through the audio processing unit 130 to inform the user that the display 733 is displayed.

An audio signal may be output through the audio processing unit 130 when a handwritten image 732 is input to the memo layer 731 and data of the handwritten image 732 is transmitted to the lower application 721. [

The user may read the appropriate feedback 733 and rewrite the handwritten image 734 on the memo layer 731 if additional questions are needed.

The memo layer 731 may transmit the sensed handwritten image 734 to the lower application 721 when the handwritten image 734 is detected again.

The sub-application 721 can compare the received handwritten image 734 with preset data to determine whether it represents an execution command or a situation in which an additional answer (feedback) is required.

When the received handwritten image 734 indicates an execution command, the control unit 160 can execute the program according to the execution command.

If it is determined that a situation requires a further answer, the sub-application 721 may send another additional answer to the memo layer 731, and the additional answer may appear on the memo layer 731.

The sub-application 721 also skips the step of comparing the received handwritten image 734 with the preset data when the received handwritten image 734 and the signal of the execute button 735 are received together, Can be executed.

The control unit 160 displays the handwritten image 734 written in the partial area of the memo layer 731 within a certain time period while the feedback 733 transmitted from the lower application 721 is displayed or the handwritten image 732 is displayed, ) Can be recognized as a reply.

The feedback 733 may be changed according to the handwritten image 732 input to the memo layer 731. [

Referring to the description between steps 610 and 620 of FIG. 6, a feedback signal indicating what the function of the lower application 721 associated with the memo layer 731 is in the partial area 738 of the memo layer 731 Can be displayed.

In some areas 738, feedback may be displayed to indicate what the handwritten image is to be input by the user.

An icon indicating what the sub-application 721 associated with the memo layer 731 may be displayed in another area 739 of the memo layer 731. [

The user can input a suitable handwriting image in a partial area of the memo layer 731 by referring to the feedback or icon displayed in some area 739 or some other area 730. [

When the gesture recognition information corresponding to the gesture input operation for reply writing is collected, the terminal 100 may process the collected gesture recognition information as information corresponding to the received handwriting image.

To be more specific, the terminal 100 may activate the function corresponding to the received handwriting image through background processing when the input of the user's reply is completed.

That is, when the function of the received handwritten image is to send a text message, the terminal 100 can execute the text message application to activate the text message function, automatically extract the sender information of the received character, It is possible to create and automatically transmit a character including information.

Meanwhile, the completion of the reply input may be a time when the electronic pen 200 is distanced from the display unit 140 by a predetermined distance, or a predetermined predefined gesture, for example, information corresponding to a period is input .

Alternatively, the terminal 100 may be displayed on the execution button 735 for the completion of the reply when the reply input starts, and the user may perform the completion of the reply input through the selection of the corresponding execution button 735.

When the handwriting image input is completed on the 730 screen, the screen of the terminal 100 may automatically return to the 720 screen, or the sleep state may be released so that the handwritten image transmission history screen may be displayed.

730 screen may vary depending on characteristics of handwritten images transmitted and received. That is, the 730 screen may be a handwritten image transmission / reception screen, or may be a chat room screen.

The memo layer 731 may be formed in a semi-transparent state on the screen 730 and the sub-application 721 located below the memo layer 731 may appear in a semitransparent manner on the memo layer 731. [

The translucency can be changed through transparency setting of the memo layer 731. [ Therefore, the transparency of the memo layer 731 can be changed, and the contrast of the displayed lower application 721 can be changed.

It is possible to execute another memo layer while the memo layer 731 is being executed in the 730 screen. The other memo layer may be interlocked with an application associated with the memo layer 731, or may interwork with another application.

The other memo layer may perform the same function as the memo layer 731.

In addition, the other memo layers may exchange data with the memo layer 731 in cooperation with each other. The received data may be sent to the sub-application 721.

The sub-application 721 may compare the received data with already stored data to select appropriate feedback.

The selected feedback may be displayed on the memo layer 731 or on the other memo layer.

When the feedback is displayed on the memo layer 731, the other memo layer may be displayed overlaid on the memo layer 731. In addition, when the other memo layer is displayed, the memo layer is semi-transparent and the memo layer 731 can be displayed through the other memo layer.

8 is a diagram for explaining a dialog between a sub-application and a memo layer according to another embodiment of the present invention.

FIG. 8 shows an example in which responses appearing on the memo layer are different depending on the type of feedback input to the sub-application and the memo layer.

The icon 811 of the contact list is selected as shown in FIG. 7 on the screen 810, the lower application 812 corresponding to the contact list is executed and displayed, and the memo layer 813 is displayed over the lower application 812 .

A feedback 817 corresponding to the character 815 can be displayed on the memo layer 813 according to the character 815 input on the memo layer 813. [

In the 820 screen and the 830 screen, other characters 825 and 835 may be input to the character 815 input to the memo layer 813, although the sub application of the 810 screen is the same.

At this time, the feedbacks 827 and 837 of the answers appearing on the memo layer 813 may be displayed differently according to the input characters 825 and 835.

This is because the controller 160 compares the characters received in the lower application with preset data, finds an appropriate answer corresponding to the character, and displays the feedback on the memo layer 813.

The 840 screen is when the 810 sub-application and other applications are running.

840 The other application shown on the screen is an application related to music.

A memo layer 841 may be displayed on top of the other application and a test or icon may be displayed on a part of the memo layer 841, for example, .

When a character 842 is input on the memo layer 841, the memo layer 841 can transmit the character 842 to the other application. The other application may compare the information of the received character with a preset test to select appropriate feedback.

The controller 160 may control the selected feedback 843 to appear on the memo layer 841.

The user can read the selected feedback 843 through the memo layer 841 displayed on the screen of the terminal and input another character 844 to the memo layer 841 when an additional answer is required.

In addition, the memo layer 841 can transmit the input other characters 844 to the other application, and the other application can compare the information of other received characters with the preset test to find other appropriate feedback.

The controller 160 may control the other appropriate feedback 845 to appear on the memorizer 841.

The user may write another character 846 in the memo layer 841 in response to another appropriate feedback 845. [

Alternatively, the user may enter some of the feedback as is in another character 846 in another appropriate feedback 845.

Alternatively, a gesture may be input to the memo layer 841 such that some feedback may be selected in other appropriate feedback 845.

The memo layer 841 may transmit the selected feedback information to the other application if some of the other suitable feedback 845 is selected by the gesture. The other application can perform execution according to the received feedback information.

For example, another suitable feedback 845, delete? play? Repeat play? When the middle play is selected, the song of the west sky, which is already inputted character 844, is played.

The method of selection may be accomplished in a variety of ways, such as by touching some characters in other appropriate feedback 845, gesturing on some of the characters, or saying some of the characters.

9 through 16 illustrate various applications using the embodiment of FIG.

9 is a view for explaining a method of sending a message to a plurality of users using a sub-application and a memo layer according to another embodiment of the present invention.

A message related application is executed in the 910 screen, and a memo layer 911 can be displayed overlaid on the message related application.

When a character 912 is input on the memo layer 911, a reply 913, which is a feedback from the message related application, may appear on the memo layer 911. [ Here, the character 912 may include an execution command and a recipient.

For example, in the character 912 of Replay Angela and Kim Sung Soo, the execution command indicates Replay, and the recipient indicates Angela and Kim Sung Soo.

The execution command may indicate an instruction to send a message to the recipient following the execution command.

After the answer 913 appears on the memo layer 911, the user can input a message 914 to be sent to the recipient in a partial area of the memo layer 911. [

The input message 914 may be sent to the message related application via the memo layer 911. [

In order to transmit the input message 914 to the recipient, the message related application may obtain the recipient's information from the data already stored in the storage unit 150. [ The message-related application may transmit the message 914 to the recipient using the acquired information of the recipient.

A message 914 may be directly transmitted to the recipient through the message related application or a message input screen 921 may be displayed as in the 920 screen so that the user of the message 914 can confirm the message. The user may modify or add a message 914 to be sent via the message input window 922. [

The user can check the contents of the message input window 922 and send the message displayed in the message input window 922 to the recipient using the send button when there is no modification or modification of the message 914 .

10 is a diagram for explaining a method of changing information of a sender using a memo layer as an application example of Fig.

The 1010 screen may be displayed over the message input screen 921 by executing the memo layer 1011 on the 920 screen.

A feedback 1012 is input to modify the sender's information on the memo layer 1011 and the control 160 receives the feedback 1012 via the memo layer 1011 and compares it with previously stored data It is possible to control the sub-application so that it can be displayed on the memo layer 1011 by searching for the feedback 1013 of the answer.

When the feedback 1013 of the answer appears in the memo layer 1011, the user can input the modification content 1014 of the sender information to a partial area of the memo layer 1011. [

When inputting the modification content 1014 of the sender information is completed, the user can select the execution button 1015 to allow the modification content 1014 to be sent to the sub-application.

In the screen 1020, the input modification information 1014 may be displayed in the transmission information input window 1021, which is a part of the display.

When the modification content 1014 appears in the sender information input window 1021, the memo layer 1011 may disappear.

The sender information input window 1021 can be displayed only when the sender information is not displayed on the screen 1020 before being modified.

Alternatively, unlike the 1010 screen, the sender information may always appear in a part of the screen.

Although FIG. 10 has been described with respect to a method for correcting sender information, it may be applied as a method for correcting recipient information or other information.

11 is a diagram for explaining a method of calling an emoticon image using a memo layer.

Referring to FIG. 11, there is a case where an emoticon image is needed during message creation. At this time, it may be inefficient if multiple steps are required to call the emoticon menu to select the required image, or if a complex selection process is required.

In order to simplify the complexity, a memo layer 1101 can be used.

For example, a gesture may be entered on the screen during message creation to invoke the memo layer 1101, and the emoticon call name 1102 may be entered on the memo layer 1101. The control unit 160 can search the emoticon images 1103 from the lower application and display them on the memo layer 1101. [

If one of the emoticons images 1103 is selected, the selected image information may be delivered to the lower application.

Also, when one of the emoticon images 1103 is selected and the execution button 1104 is selected, the selected emoticon image information can be delivered to the lower application.

The selected emoticon image may be inserted into a portion of the message being created.

11 illustrates an example of selecting an emoticon image during message creation, it can be applied to various message related applications such as a kakao chat.

FIG. 12 is a diagram illustrating an example in which when a part of a mail is empty when a mail is sent, for example, in a state in which a content of some input items among input items requiring input for sending a mail is not inputted, Fig. 2 is a diagram for explaining a method of giving feedback to a user.

1210 When the mail sending is executed on the screen, the memo layer 1221 interlocked with the mail application can review the contents inputted on the mail application.

When the mail sending is executed on the screen 1210, the memo layer 1221 examines the contents and if the area where the subject is inputted or the area where the body (body) is input is empty, that is, The memo layer 1221 is displayed as shown on the screen 1220 and the warning feedback can be displayed on the memo layer 1221. [ For example, the warning feedback can be Send this msg without text?

The user can input a handwritten image corresponding to the input content to be input in the input area of the subject or the input area of the body in the memo layer 1221. [ At this time, the memo layer 1221 checks whether the handwritten image exists in the input area of the subject or the input area of the body, and if not, the handwritten image can be transferred to the mail application.

The memo layer 1221 sends the input handwritten image to the mail application, and the mail application can send the mail by adding the handwritten image received.

In addition, if the user does not have a content to be added to the topic input area or the body input area, the user can input characters or symbols such as no and X, which are pseudo marks that are not needed in a part of the memo layer 1221.

When the memo layer 1221 receives a request that the unnecessary information is not needed, the memo layer 1221 can send the mail through the mail application with the input area being empty.

13 is a diagram showing a method of registering a web page bookmark using a memo layer.

A memo layer 1312 may be executed to bookmark the site while the web page 1311 searches for the site on the screen 1310 and displayed on the web page 1311 in a superimposed manner.

When the bookmark command 1314 is input on the memo layer 1312, the memo layer 1312 can send the bookmark command 1314 to the bookmark application.

The bookmark application may bookmark an Internet address 1313 representing the web page 1311 according to the received bookmark command 1314. [

The bookmark application may send an answer 1316 of a question 1315 as to whether the address of the bookmark is correct on the memo layer 1312 or what the name of the bookmark is to be made.

The bookmark application may also send a question answer 1317 to the memo layer 1312 asking which folder to store after receiving the bookmark name 1316 information.

The user can input information on the name 1317 of the folder to be stored as the answer in the partial area of the memo layer 1312 after viewing the answer 1317 of the question.

The bookmark application can bookmark the Internet address 1313 to the folder of the name 1318 when a predetermined time elapses after the name 1318 of the folder is input to the memo layer 1312. [

If the name 1318 of the folder is input to a partial area of the memo layer 1312 and the execution button 1319 is selected, the bookmark application may bookmark the Internet address 1313 to the folder of the name 1318 .

The 1320 screen shows a result (1321) in which the Internet address 1313 is bookmarked according to the bookmark command executed on the 1310 screen.

The user can check whether the desired Internet address 1313 is properly bookmarked on the screen 1320. If there is an error, the user can execute the memo layer 1312 again to re-execute the bookmark.

14 is a diagram illustrating a method of sharing a web page using a memo layer with another user.

When the user wants to share the web page 1411 with another user while searching the web page 1411 on the screen 1410, the user can execute the memo layer 1412 to be displayed on the web page 1411 in a superimposed manner have.

The user may enter a share command 1412, e.g., Share Page, in a portion of the memo layer 1412. [ The memo layer 1412 may send the information of the sharing command 1412 to the sharing application.

The shared application may send information 1414 of the apps to which the web page 1411 may be shared on the memo layer 1412 based on information of the received sharing command 1412. [

The user can select the number of apps 1414 listed in front of the apps.

The shared application corresponding to the number may be executed when a predetermined time elapses after the number is input to the memo layer 1412. [

Also, when the number is input to a partial area of the memo layer 1412 and the execution button 1318 is selected, the shared application corresponding to the number can be executed.

The user selects the number on the screen 1410 and inputs the selected number to a partial area of the memo layer 1412. However, instead of selecting the number on the screen 1420, the user can touch the desired application through the screen scroll function 1421, .

15 is a diagram illustrating a method of sorting files in a My Folder using a memo layer.

The memo layer 1502 may be executed in a state where the My Folder application 1501 is executed and overlapped and displayed.

When the sorting command 1503, for example, list by, is input to a partial area of the memo layer 1502, the My Folder application 1501 associated with the memo layer 1502 can receive the information of the sort command 1503 have.

In accordance with the received information, the My Folder application 1501 may send an option 1504 of sorting on the memo layer 1502.

If at least one of the sorting options 1504 is selected, for example, when the Type is input, the files stored in the My Folder may be sorted according to the selected option.

At least one of the sorting options 1504 is selected and the files can be sorted according to the selected option when a certain time has elapsed.

Also, if at least one of the sorting options 1504 is selected and the execute button 1506 is selected, the files may be sorted according to the selected option.

16 is a diagram illustrating a method of arranging images in a gallery in which picture files are stored using a memo layer.

The memo layer 1602 may be executed in a state in which the gallery application 1601 is executed and overlapped and displayed.

When the sorting command 1603, for example, view by is input to a partial area of the memo layer 1502, the gallery application 1601 associated with the memo layer 1602 can receive the information of the sort command 1603 .

Depending on the received information, the gallery application 1601 may send an option 1604 of sorting on the memo layer 1602.

If at least one of the sort options 1604 is selected, for example, 3, the picture files stored in the gallery can be sorted according to the selected option.

At least one of the sorting options 1604 is selected and the picture files may be sorted according to the selected option when a predetermined time has elapsed.

Also, if at least one of the sorting options 1604 is selected and the execute button 1606 is selected, the picture files may be sorted according to the selected option.

The memo layer 1602 is formed in a translucent manner so that the gallery application 1601 located below the memo layer 1602 can appear in a semitransparent manner in the memo layer 1602. [

17 is a diagram showing a method of setting an alarm using a memo layer.

The memo layer 1702 may be executed in a state in which the alarm application 1701 is executed and overlapped and displayed.

The alarm application 1701 associated with the memo layer 1702 sets the information of the command 1703 of the setting in the memo layer 1702 .

According to the received information, the alarm application 1701 may send the set information 1704 on the memo layer 1702. [

Thus, the user can confirm the set information 1704.

If there is an error in the set information 1704, the memo layer 1702 may display an error message so that the setting command 1703 can be reentered in a part of the memo layer 1702. [

For example, the input set information 1704 in some areas of the memo layer 1702 may indicate feedback requesting re-entry because of an error.

18 is a diagram illustrating a method of registering an account using a memo layer.

The memo layer 1802 may be executed while the accounting application 1801 is executed, so that the memo layer 1802 may be overlaid.

The account application 1801 associated with the memo layer 1802 can receive the information of the register command 1803 when the register command 1803 is input to a partial area of the memo layer 1802. [

The command 1803 of the registration may be information of a web page, for example, Twitter.

When the account application 1801 is executed and the memo layer 1802 is overlaid on the account application 1801 and the information of the web to be registered is input in a partial area of the memo layer 1802, It is possible to execute a process that can be registered in the web.

According to the received registration command 1803, the account application 1801 may send an answer 1804 of an additional question required for registration on the memo layer 1802.

Additional information 1805 according to the answer 1804 of the additional question may be input to a partial area of the memo layer 1802. [

The account application 1801 may review the input additional information 1805 received through the memo layer 1802 and determine that additional information other than the input additional information 1805 is not needed.

If it is determined that the additional information is not needed, the account application 1801 may send the information of the character 1806 indicating that the registration is completed on the memo layer 1802.

The specific application being executed can be controlled using the memo layer in a state where the specific application is being executed in the terminal 100 according to another embodiment of the present invention.

For example, the terminal 100 can monitor whether or not the memo layer is called by the user while the music playback application is being executed as a sub-application.

When the memo layer is called by the user, the terminal 100 can activate the menu layer having a title and an input menu button set on the screen. That is, the menu layer in which the title and the input menu button are set can be displayed so as to overlap with the screen in which the music playback application is running. For example, the note layer title might say, "Please write a song to play!"

The terminal 100 can write a desired content (for example, "me ") in the memo layer and monitor whether the displayed input menu button is pressed.

When the terminal 100 detects that a signal, for example, an input display button, is pressed by the user, the terminal 100 recognizes that the content written by the user is " To the currently running music playback application.

The music playback application can search for a song titled "I do not care" and reproduce the song identified by the search. At this time, the range to search for the song may be designated by the setting of the user. For example, a search target may be a song that is stored in the terminal 100, or a web site that provides a music service.

In order to set the web site as a search target, information for authentication of the web site or the like needs to be managed by a terminal or input by a user.

If the search result is a plurality of search results, a plurality of songs corresponding to the search result may be sequentially reproduced or a song to be reproduced by the user may be selected. For example, in order to select a song to be played by a user, it is desirable to display the searched result in the form of a list on the screen.

In addition, when the handwritten image of the received "heartbeat" can not be found in the already set data, the terminal 100 can send feedback to the memo layer so that the feedback can be displayed on the screen.

The user can input another handwritten image corresponding to the feedback to a partial area of the memo layer.

The control unit 160 of the terminal 100 may recognize the other handwritten image and then send the information of the other handwritten image to the music reproduction application to control the instruction of the other handwritten image to be executed.

The specific application being executed can be controlled using the memo layer in a state where the chat message application is executed in the terminal 100 according to another embodiment of the present invention.

The controller 160 of the terminal 100 may control the chat message function to activate as background processing. The control unit 160 may control the automatic transmission of the gesture recognition information input while the message is being output to the screen, as a response message to the received chat message.

In this process, the control unit 160 can control to open a chat room including the received chat message, or to write the received message and the response message in the chat room opened in advance in association with the sender of the received chat message.

The control unit 160 can support sending of reply e-mails in the same manner as in the text message service even in the case of e-mail. That is, when prompt reply transmission to the received e-mail message is required, the control unit 160 extracts the sender e-mail information of the e-mail displayed in the message, and includes gesture recognition information input by the user as contents information of the e-mail, As shown in FIG.

As described above, the method for operating the electronic device and the electronic device according to the embodiment of the present invention can be applied to the electronic pen 200 and the electronic pen 200 so that the user can perform more various and convenient function operations in an environment using the electronic pen 200 (Not shown) to be mapped.

Accordingly, the user can perform at least one of the type of the function currently being activated, the electronic pen state information, and the gesture recognition information by associating them with each other or independently operating the functions.

Meanwhile, the terminal 100 may further include various additional modules according to the provided mode. That is, the terminal 100 includes a short-range communication module for short-range communication, a wired communication method for the short-range communication or an interface for data transmission / reception using the wireless communication method, an Internet A communication module, and a digital broadcasting module that performs digital broadcasting reception and reproduction functions.

These components can not be enumerated because of a wide variety of variations depending on the convergence trend of the digital device, but it is also possible that components having the same level as the above-mentioned components are further included in the device have.

Also, it is needless to say that the terminal 100 of the present invention may be excluded from the specific configurations in the above configuration or may be replaced with other configurations in accordance with the provision mode thereof.

Which will be readily apparent to those skilled in the art.

In addition, the terminal 100 according to the embodiment of the present invention may include all mobile communication terminals operating based on communication protocols corresponding to various communication systems, a PMP (Portable Multimedia) Player, a digital broadcasting player, a PDA (Personal Digital Assistant), a music player (for example, an MP3 player), a portable game terminal, a smart phone, a notebook, a handheld PC, And an application device therefor.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. , And are not intended to limit the scope of the present invention.

It will be apparent to those skilled in the art that other modifications based on the technical idea of the present invention may be practiced without departing from the scope of the invention disclosed herein.

100: terminal 110:
120: input unit 140:
160: control unit 200: electronic pen
510: Lower application drive engine 520: Memo layer drive engine

Claims (17)

A method of controlling an application of an electronic device including a touch screen,
The application being executed and displayed on the touch screen;
Detecting a predetermined gesture on the touch screen during execution of the application;
Wherein the gesture is superimposed on the application,
A handwriting input area for displaying data to be provided to the application;
Displaying a memo layer including a display area on which data provided from the application is displayed;
Receiving a handwriting image in the handwriting input area of the memo layer;
Comparing the input handwritten image with recognized data;
And displaying feedback from the application on the display area of the memo layer according to the result of the comparison.
The method according to claim 1,
Wherein the handwritten image and the feedback can be composed of a plurality of features and the feedback corresponding to the handwritten images appearing in the handwriting input area are displayed in the different display areas respectively.
The method according to claim 1,
Wherein the memo layer is semitransparent and is displayed overlaid on a screen in which the application is running, the application includes a feature displayed through the memo layer.
The method according to claim 1,
Wherein the memo layer comprises a plurality of memo layers, the memo layers being characterized in that the plurality of memo layers are superimposed and displayed in multiple layers in accordance with feedback received from the application
The method of claim 1, wherein
Wherein the feedback can be composed of a plurality of features and includes a feature appearing in a partial area of the memo layer.
6. The method of claim 5,
The plurality of feedbacks may be formed by an instruction for executing the application, and when a partial region in which the plurality of feedbacks is displayed is selected, the plurality of instructions in the selected partial region are executed, Control method.
The method according to claim 1,
When another application is executed instead of the application,
And another memo layer linked with the other application is executed.
8. The method of claim 7,
Wherein the feedback displayed on the different memo layer is differently displayed according to a function of the another application.
The method according to claim 1,
Wherein the handwritten image and the feedback are sequentially displayed on a partial area of the memo layer.
The method according to claim 1,
Wherein the handwritten image and the feedback are interactive.
The method according to claim 1,
Wherein the handwritten image or the feedback is at least one of a character, a feedback, an image, a pattern, a number, a special character, and an email address.
An electronic device comprising a touch screen,
An application is executed and displayed on the touch screen,
Handwriting recognition, overlapped and displayed in the application
A handwriting input area for displaying data to be provided to the application;
A memo layer including a display area in which data provided from the application is displayed,
A display unit for receiving a handwritten image in the handwriting input area of the memo layer and displaying the handwritten image in the handwriting input area;
Detecting a predetermined gesture on the touch screen during execution of the application,
The input handwritten image is recognized and compared with preset data,
And a control section for controlling the feedback from the application to appear in the display area of the memo layer according to the comparison result.
13. The method of claim 12,
Wherein the handwritten image and the feedback can be composed of a plurality of, and the feedback corresponding to the handwritten images appearing in the handwriting input area are displayed in different display areas, respectively.
The method of claim 12, wherein
Wherein the feedback can be composed of a plurality of controls, and controls to appear in a part of the memo layer.
15. The method of claim 14,
Wherein the plurality of feedbacks may be formed of an instruction for executing the application and a control unit for controlling the plurality of instructions in the selected partial area to be executed when a partial area in which the plurality of feedbacks is displayed is selected, .
13. The method of claim 12,
When another application is executed instead of the application,
And a control section for controlling execution of another memo layer interlocked with the other application.
17. The method of claim 16,
And the control unit controls the feedback displayed on the different memo layer to be displayed differently according to the function of the other application.

Images