KR101898162B1 - Apparatus and method of providing additional function and feedback to other apparatus by using information of multiple sensor - Google Patents

Abstract

The present invention relates to an apparatus and method for providing additional functions and feedback to other apparatuses through multi-sensor detection. The first apparatus includes a sensor input unit for sensing user input through an unstructured user interface region, An operation recognition unit that recognizes a device control operation corresponding to the sensed user input and generates a device control signal corresponding to the recognized device control operation; 2 device, the second device receiving the device control signal and operating in response to the received device control signal, wherein the structured user interface is inactive with the structured user interface inactive, And provides the function of the interface based on the device control signal.

Description

[0001] APPARATUS AND METHOD OF PROVIDING ADDITIONAL FUNCTION AND FEEDBACK TO OTHER EQUIPMENT WITH MULTIPLE SENSOR DETECTION [0002]

A technology that provides additional functions to other devices so that users can perform tasks related to productivity more efficiently and effectively by using a user's device such as a smart phone, will be.

Along with the popularization of tablet PCs with a large screen touch screen, the development of computing technology has enabled users to produce contents without having to spend on content any more. However, due to the lack of hardware input devices such as a mouse and a keyboard, the tablet PC is not equipped with professional productivity. In order to overcome these limitations, devices having keyboards and track pads and tablet PCs capable of being detachable . However, the hardware combination with these input devices increases the weight and size of the device, resulting in poor mobility.

Most of the applications provided by the existing touch screen devices are different in the detailed UI (User Interface) configuration, but have a large commonality that they have a structured UI on one screen in order to provide additional functions basically. For example, a drawing or handwriting application provides a palette GUI (Graphical User Interface) that allows you to select a color, or a GUI that lists a variety of tools to allow the user to switch functions. In addition, the application for editing images also provides a variety of tools included in the structured UI as graphics for the user to change the color of the photo or to provide various effects. However, these GUI-based functional switching methods have three major limitations.

In order to use additional functions, these function switching methods are used to stop an operation performed by a user and to make an interactive object (an interactive object such as a button or an icon) in a structured UI performing a desired function I have to find it. This step interferes with the performance of the task because the user stops performing the task.

Next, in order for the user to switch the functions, input operations such as touching a desired object found by eyes (in the case of a touch screen device) or clicking (in the case of a general PC such as a desktop) must be performed.

Finally, the structured UI of most existing applications is defined to allow users to access additional functionality multiple times for efficient use on a single screen. For example, in order to switch the color while the user is using the handwriting application, the palette icon should be touched and then the palette window should be touched again.

These operations cause the user's creative activities to be interrupted in the middle, which may result in usability inhibition, user experience hindrance, and overall work efficiency.

Korean Patent No. 0986619 "Multi Input / Output Device and Method Using Mobile Communication Terminal" Korean Patent Laid-Open No. 2011-0107143 entitled " Method and Apparatus for Controlling Function of Portable Terminal Using Multi-Input "

According to one embodiment, it is possible to reduce unnecessary time for the user, thereby reducing the process of searching for a desired function.

According to one embodiment, it is possible to reduce the process of selecting a function such as a user unnecessarily touching multiple times.

According to one embodiment, it is possible to increase the efficiency of task execution by the user and to improve the user experience (UX) through quick switching of functions.

The first device according to an embodiment of the present invention includes a sensor input unit for sensing a user input through an unstructured user interface area, a device recognition unit for recognizing a device control operation corresponding to the sensed user input, And a data communication unit for transmitting the generated device control signal to a second device providing a structured user interface, wherein the second device receives the device control signal, And a function of the structured user interface is provided based on the device control signal in response to the received device control signal while the structured user interface is inactive.

The second device according to an embodiment includes a structured user interface area including at least one or more indicator user interfaces, wherein at least one of the unstructured user interface area and the structured user interface area comprises at least one of an icon, , And a color value.

The second device according to an embodiment operates with a function corresponding to the received device control signal and changes a previous setting value to a setting value corresponding to the device control signal.

The second device according to an exemplary embodiment of the present invention operates as a function corresponding to the received device control signal and selects a specific additional function corresponding to the device control signal among the additional functions related to the application driven by the second device And drives the motor.

The data communication unit may receive a second device control signal corresponding to a second device control operation collected by the second device, and the operation recognition unit may receive an additional function corresponding to the received second device control signal Lt; / RTI >

The first device according to an embodiment further includes a control unit for driving the recognized additional function.

The control unit controls the content generated by the driven additional function to be provided to the second device through the data communication unit.

The control unit controls the second device to receive and output feedback corresponding to the device control signal.

The sensor input unit senses at least one of a pressure, a frequency, a speed, an acceleration, a direction, a rotation, a sound, and a number of the user input. The operation recognition unit senses the sensed pressure, The device control operation is recognized according to at least one of speed, acceleration, direction, rotation, sound, and number.

The second device according to an embodiment includes a data communication unit for receiving a device control signal transmitted from the first device, and a data communication unit operable in response to the received device control signal, wherein, when the structured user interface is inactivated, And a control unit for controlling an application to provide a function of an interface based on the device control signal, wherein the first device senses a user input through an unstructured user interface area, and the device corresponding to the sensed user input The control unit recognizes the control operation, generates a device control signal corresponding to the recognized device control operation, and transmits the generated device control signal to the data communication unit.

The control unit controls the application by changing a previous setting value to a setting value corresponding to the device control signal.

The control unit controls the application and drives a specific additional function corresponding to the device control signal among the additional functions related to the application currently being operated.

The data communication unit according to an embodiment receives content generated by the additional function driven by the first device, and the control unit outputs the received content through the currently-running application .

A method of operating a first device according to an embodiment includes sensing a user input through a non-structured user interface area in a sensor input unit, recognizing a device control operation corresponding to the sensed user input in an operation recognition unit Generating a device control signal corresponding to the recognized device control operation and transmitting the generated device control signal to a second device providing a structured user interface in the data communication unit, 2 device receives the device control signal and operates in response to the received device control signal to provide the function of the structured user interface based on the device control signal while the structured user interface is inactive .

The second device according to an embodiment operates with a function corresponding to the received device control signal and changes a previous setting value to a setting value corresponding to the device control signal.

The second device according to an exemplary embodiment of the present invention operates as a function corresponding to the received device control signal and selects a specific additional function corresponding to the device control signal among the additional functions related to the application driven by the second device And drives the motor.

The method of operating the first device according to an embodiment may further include the steps of receiving, in the data communication unit, a second device control signal corresponding to a second device control operation collected by the second device, Recognizing an additional function corresponding to the received second device control signal, and, in the control unit, driving the recognized additional function.

The step of driving the recognized additional function according to an embodiment includes controlling the content generated by the driven additional function to be provided to the second device through the data communication unit.

The step of driving the recognized additional function according to an embodiment includes controlling to receive and output feedback corresponding to the device control signal from the second device.

Sensing the user input in accordance with an embodiment includes sensing at least one of pressure, frequency, speed, acceleration, direction, rotation, sound, and number of user inputs to the user, The step of recognizing the device control operation corresponding to the input includes recognizing the device control operation according to at least one of the sensed pressure, frequency, speed, acceleration, direction, rotation, sound, and number.

According to the present invention, it is possible to reduce a process of searching for a desired function by reducing unnecessary time for a user.

According to the present invention, it is possible to reduce a process of selecting a function such as a user unnecessarily touching several times.

According to the present invention, it is possible to increase the efficiency of task execution by the user and to improve the user experience (UX) through quick switching of functions.

1 is a diagram illustrating an overall system according to one embodiment.
2 is a view for explaining a first device and a second device according to an embodiment of the present invention.
3 to 7 are diagrams for explaining an embodiment in which the first device is controlled to change the input mode of the second device.
Figs. 8 and 9 are views for explaining an embodiment in which the first device is controlled to perform the additional function of the second device. Fig.
10 and 11 are views for explaining an embodiment in which the second device is controlled to perform the additional function of the first device.
12 to 14 are diagrams for explaining an embodiment in which at least one first device is used as an input device of the second device.
15 to 18 are diagrams for explaining an embodiment for controlling the second device by checking the display device user interface and the context information displayed on the first terminal.
FIG. 19 is a view for explaining an embodiment of detecting the change of the touch generated in the first device, changing the context information, and controlling the second device based thereon.
20 is a view for explaining an embodiment for providing at least one feedback from visual feedback, tactile feedback, and auditory feedback in a second device in accordance with an operation performed by the first device;
21 is a view for explaining another embodiment for providing at least one feedback from visual feedback, tactile feedback, and auditory feedback in a second device in accordance with an operation performed by the first device.
22 is a view for explaining an operation method of the first device according to the embodiment.

It is to be understood that the specific structural or functional descriptions of embodiments of the present invention disclosed herein are presented for the purpose of describing embodiments only in accordance with the concepts of the present invention, May be embodied in various forms and are not limited to the embodiments described herein.

Embodiments in accordance with the concepts of the present invention are capable of various modifications and may take various forms, so that the embodiments are illustrated in the drawings and described in detail herein. However, it is not intended to limit the embodiments according to the concepts of the present invention to the specific disclosure forms, but includes changes, equivalents, or alternatives falling within the spirit and scope of the present invention.

The terms first, second, or the like may be used to describe various elements, but the elements should not be limited by the terms. The terms may be named for the purpose of distinguishing one element from another, for example without departing from the scope of the right according to the concept of the present invention, the first element being referred to as the second element, Similarly, the second component may also be referred to as the first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Expressions that describe the relationship between components, for example, "between" and "immediately" or "directly adjacent to" should be interpreted as well.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises ", or" having ", and the like, are used to specify one or more of the features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

1 is a diagram illustrating an overall system 100 in accordance with one embodiment.

The overall system 100 according to one embodiment includes a first device 110 and a second device 120.

In this specification, the first device 110 includes a touch screen such as a smart phone and includes at least one sensor module such as an acceleration sensor, a gyro sensor, a proximity sensor, an image sensor (e.g., a camera), a microphone, It can be interpreted as an equipped device. Also, the second device 120 may not be provided with various sensor modules such as a tablet, but may be interpreted as a device having a communication module capable of communicating with the first device 110 on a large screen.

The touch screen in the first device 110 essentially includes a non-structured user interface, and a structured user interface may optionally exist. If a structured user interface exists, the structured user interface includes one or more interactive objects (IO), such as text / images, which can be used to use the manipulation of the first device 110 itself. On the other hand, the unstructured user interface of the first device 110 may perform various functions.

For example, the unstructured user interface of the first device 110 may perform a function of accepting a user's input operation to perform a command such as providing, switching, etc., additional functions to the second device 120 have. At this time, the user's input operation can be sensed through the touch sensor in the touch screen.

In addition, the unstructured user interface of the first device 110 may perform a function of providing visual feedback when the additional function of the second device 120 is provided and switched. The sensor module in the first device 110 performs a function of detecting various conditions of the user in addition to the touch input operation of the user through the unstructured user interface of the touch screen.

In one embodiment, the second device 120 is illustrated as a tablet PC, but may be connected to the first device 120, such as a TV, a notebook PC, a desktop PC, a tablet PC, and a smart phone, And may be a variety of devices equipped with a screen. Like the first device 110, the user interface of the second device 120 may include a structured user interface in addition to the unstructured user interface, which may be configured differently depending on the application. The structured user interface includes one or more objects similar to the first device 110, and each of the objects serves to operate the second device 120 itself. The unstructured user interface of the second device 120 plays a role of visually outputting the feedback while executing the command value received through the first device 110.

In order to implement the present invention, a first device having a touch screen is assumed as a basic premise.

Both the first device 110 and the second device 120 must have a data transmission / reception module. For this purpose, the first device 110 and the second device 120 transmit / receive data to / from each other. The method used at this time may be a state in which they are connected to the same Wi-Fi and share a local network, Data transmission / reception system. The communication method between the two devices can be applied variously according to the manufacturer or software development.

2 is a view for explaining the first device 210 and the second device 220 according to an embodiment of the present invention.

The first device 210 according to one embodiment includes a sensor input unit 211, a touch screen input unit 212, an operation recognition unit 213, a control unit 214, an output unit 215, and a data communication unit 216 .

The second device 220 according to one embodiment includes a data communication unit 221, a comparison operation unit 222, a user setting storage unit 223, a user input recognition unit 224, a control unit 225, Lt; RTI ID = 0.0 > 226 < / RTI >

The second device 220 may be a device that provides direct output to a user's input (e.g., a general desktop PC, a TV, a smartphone, a tablet, etc.), provided that the first device 210 and the second device 220 transmit and receive data. PC). In addition, the first device 210 functions to convert and convert the output of the second device 220. The touch screen input unit 212 may collect the first touch screen device 210 together with the sensor input unit 211 when the user performs a specific touch gesture and various sensors. In particular, the sensor input unit 211 can recognize the device control operation corresponding to the sensed input through the unstructured user interface area, and generate the device control signal corresponding to the recognized device control operation. To this end, the sensor input unit 211 may sense at least one of pressure, frequency, speed, acceleration, direction, rotation, sound, and number of user inputs.

The operation recognition unit 213 of the first device 210 may classify and decide the corresponding operation. That is, the operation recognition unit 213 recognizes the device control operation corresponding to the sensed input, classifies and decides the operation, and generates a device control signal corresponding to the recognized device control operation can do. For example, the motion recognition unit 213 may recognize the device control operation according to at least one of the sensed pressure, frequency, velocity, acceleration, direction, rotation, sound, and number.

The generated device control signal is transmitted to the second device 220 providing the structured user interface through the data communication unit 216.

The second device 220 may receive the device control signal and may operate in response to the received device control signal to provide the functionality of the structured user interface based on the device control signal with the structured user interface inactive have. To this end, the second device 220 may comprise a structured user interface area comprising at least one or more indicator user interfaces. Also, at least one of the unstructured user interface area and the structured user interface area may output context information in at least one form of icons, text, and color values.

The controller 214 according to an exemplary embodiment may be interpreted as a component that controls the overall operation of the first device 210 by receiving commands such as a processor.

Also, the result value of the recognized operation can be fed back to the user through the output unit 215 through the visual, auditory, and tactile senses that an input event has been recognized.

The recognition result value transmitted to the data communication unit 221 of the second device 220, that is, the device control signal, may implement the additional function provided by the application of the second device 220.

According to one embodiment, the second device 220 may operate with a function corresponding to the received device control signal, and may change the previous setting value to a setting value corresponding to the device control signal.

To this end, the device control signal is transmitted to a comparison operation unit 222 that matches the operation previously set by the user in the user setting storage unit 223 and the function corresponding to the operation. If there is a matching result between the recognition value of the first device 210 and the user setting through the matching, the control unit 225 may be provided with a function for the corresponding setting value. If the function for the set value is automatically executed without additional input by the user, the control unit 225 can output the function through the output unit 226. [ In one example, the control unit 225 may operate in response to the received device control signal, but may control the application to provide the functionality of the structured user interface based on the device control signal, with the structured user interface inactive. In addition, the control unit 225 may control the application, and may change the previous setting value to a setting value corresponding to the device control signal to control the application.

In addition, the control unit 225 may control the application, and may drive a specific additional function corresponding to the device control signal among the additional functions related to the currently-running application.

The data communication unit 221 receives the content generated by the additional function driven by the first device 210 and the control unit 225 can output the received content through the currently running application.

This corresponds to the embodiment of Figures 8 and 9.

Next, when the user input is input to the user input recognizing unit 224, the converted function may be provided to the user through the output unit 226 when the converted function for the user's additional input is executed.

This corresponds to the embodiment disclosed in Figs. 3 to 6.

The user may be provided feedback in the form of visual, auditory or tactile information to intuitively inform the user of the provision of the additional function of the second device 220 through the first device 210. [

In order to provide feedback to the user in the form of visual information, the first device 210 and the second device 220 may operate as corresponding functions when receiving the device control signal. In this case, the first device 210 or the second device 220 may provide contextual information and an indicator corresponding to the device control signal.

The first device 210 includes an unstructured user interface area of the touch screen as described above. This unstructured user interface can output context information of icon (image), text and color value types corresponding to the device control signal.

And the second device 220 can display the indicator user interface through the output unit 226. [ The indicator user interface can similarly output context information in the form of icons, text, and color values.

In addition to visual feedback, the auditory feedback may selectively output to the output 215 or output 226 of the first device 210 or the second device 220. For example, when the provision and switching of the additional function is completed, a notification sound may be output from both the speaker of the first device 210 and the speaker of the second device 220, and the speaker of the first device 210 or the speaker of the second device 220 Only one of them may sound a beep.

In the case of providing tactile feedback, it is also possible to output to both of the two devices or to one of the two devices in the same manner as the previous voice feedback through an actuator such as a vibration motor in the first device 210 and the second device 220 .

The prior visual / auditory / tactile feedback may also be provided in combination of two or more.

For example, the second device 220 may be operated with a function corresponding to the received device control signal, and may be operable to drive a particular additional function corresponding to the device control signal, among additional functions associated with applications running on the second device have.

For example, the first device 210 receives a second device control signal corresponding to a second device control operation collected in the second device 220 through the data communication unit 216, and the operation recognition unit 213 It is possible to recognize the additional function corresponding to the device control signal for the received second device. At this time, the control unit 214 may drive the recognized additional function through the first device 210.

The control unit 214 may control the content generated by the driven additional function to be provided to the second device 220 through the data communication unit 216. The control unit 214 can control the second device 220 to receive and output feedback corresponding to the device control signal.

3 to 7 are diagrams for explaining an embodiment in which the first device is controlled to change the input mode of the second device.

Referring to FIG. 3, the user 310 may change the input means by controlling the first device during the input with the 'black pen' through the second device. That is, as shown at 320, the user may touch and hold only two fingers on the smartphone to switch the 'black pen' input to the 'red pen' input.

That is, the first device can sense the operation of touching the two fingers with the smartphone through the sensor, and recognize the corresponding device control operation. If the device control operation corresponding to the operation of touching the two fingers with the smartphone is predefined as the 'red pen' input, the color may be changed from the existing color to red when the user touches the two fingers to the first device.

As with reference numeral 330, the user may use the 'red pen' input according to the changed color and use the changed input again to the color before red when the touch of two fingers is released as shown at 340.

Referring to FIG. 4, the user 410 can change the input means of the second device to the 'brush tool' by controlling the first device during the input with the 'ball pen' through the second device. That is, as shown at 420, the user may touch the smartphone with only two fingers to turn the 'ball pen' input into a 'brush tool' input.

That is, the first device can sense the operation of touching the two fingers with the smartphone through the sensor, and recognize the corresponding device control operation. If the device control operation corresponding to the operation in which two fingers are touched on the smartphone is pre-set as the 'brush tool' input, the existing tool may be changed to the current tool when the user touches the first device with two fingers.

As with reference numeral 430, the user may use the 'Brush tool' input according to the changed tool, and if the two fingers are touched, as shown at reference numeral 440, .

Referring to FIG. 5, the user 510 can change the input means of the second device to the 'tool for cutting' by controlling the first device during the input with the 'ball pen' through the second device. That is, as shown at reference numeral 520, the user can touch and hold only two fingers on the smartphone to convert the 'ball pen' input to the 'scissor tool'.

That is, the first device can sense the operation of touching the two fingers with the smartphone through the sensor, and recognize the corresponding device control operation. When the device control operation corresponding to the operation of touching two fingers to the smartphone is pre-set with the 'scissor tool', the user can change from the existing tool to the current tool when the user touches the first device with two fingers.

As with reference numeral 530, the user may use the 'scissor tool' input according to the changed tool, and if the two fingers are touched off as shown at reference numeral 540, the tool that has been changed to the 'scissor tool' .

Referring to FIG. 6, the user of 610 may change the input means of the second device to the 'brush tool' input by touching the first device with a 'ball pen' during the input through the second device. Also, the user can change the input means of the second device to the 'red pen' input through two finger touches. Further, the user may change the input means of the second device to a "clipping tool" through a three-finger touch.

Referring to FIG. 7, the user at 710 may control the first device during use of the 'image editing tool' through the second device, and change the input means of the second device to 'mosaic tool'. That is, as shown at reference numeral 720, the user can keep one finger touched on the smartphone and switch the input using the 'mosaic tool' during the use of the 'image editing tool'.

In addition, the user senses the operation of touching the two fingers with the smartphone through the sensor as shown by reference numeral 730, and can change the use of the 'mosaic tool' to the use of the 'scissor tool'.

Figs. 8 and 9 are views for explaining an embodiment in which the first device is controlled to perform the additional function of the second device. Fig.

Referring to FIG. 8, as shown at 810, a user may perform touch input to the first device during use of the ballpoint pen to execute an additional function through an application executable in the second device. As shown at 820, when the user performs one finger touch to the first device, a 'photo album' window may be popped up among applications executable on the second device. As shown at 820, the user can perform an additional function of selecting and inserting a desired image through the pop-up 'photo album' window.

Referring to FIG. 9, as shown at reference numeral 910, a user can input an input using the two fingers to the first device during use of the ballpoint pen tool, thereby executing an additional function through an application executable in the second device. As shown in reference numeral 920, when the user inputs a pinch-in using two fingers to the first device, it is possible to output 'entire notebook and sketch library' among the applications executable in the second device . As shown at 920, the user may perform an additional function of selecting and inserting a desired image through the pop-up 'All Note and Sketch Library' window.

10 and 11 are views for explaining an embodiment in which the second device is controlled to perform the additional function of the first device.

Referring to FIG. 10, a user can perform an input for executing an additional function in a first device during drawing and editing a document through a second device, as shown at reference numeral 1010.

As shown in reference numeral 1020, when the user touches the second device using one finger, an additional function through the application executable in the first device can be executed. As shown in reference numeral 1020, when a user inputs a touch using a finger to the second device, a 'recorder' may be executed among applications that can be executed in the first device, as shown at reference numeral 1030. As shown at reference numeral 1040, the user can record a desired voice through the executed 'recorder'. In addition, the recorded voice may be delivered in text form to the second device via the Voice to Text function of the first device. Thus, as shown at 1050, the second device can output the transmitted text. The Voice to Text function may be executed by the first device, but may be executed by receiving the voice from the second device.

Referring to FIG. 11, the user may perform an input for executing an additional function in the first device during drawing and document editing through the second device, as shown at reference numeral 1110.

As shown in reference numeral 1120, when the user performs a double-touch input to the second device using one finger, the additional function through the application executable in the first device can be executed. As shown in reference numeral 1120, when the user inputs a touch using the finger to the second device, the 'camera' can be executed among the applications executable in the first device, as shown at reference numeral 1130. As shown at reference numeral 1140, the user can shoot a desired image through the executed 'camera'. Further, the photographed image can be transmitted to the second device. Thus, as shown at reference numeral 1150, the second device can output the transmitted image.

12 to 14 are diagrams for explaining an embodiment in which at least one first device is used as an input device of the second device.

As shown in FIG. 12, the user can control the function of the moving picture player using the first device during playback of the moving picture player through the second device. Specifically, as shown at reference numeral 1210, the user can touch input the first device and move the first device. At this time, the first device senses the touch and movement from the user, and transmits the sensed touch and movement to the second device to perform an additional function (video player control) to the second device.

Similarly, as shown in reference numeral 1220, the user can touch input the first device and move the first device during game play requiring multiple operations. In this case, the first device senses the touch and movement from the user and can transmit the sensed touch and movement to the second device. The second device receives the device control signal transmitted from the first device and transmits the additional function (game operation) to the second device Can be performed.

13, an embodiment 1300 for controlling a second device through a plurality of smartphones, among game playing in a multiplayer mode, will be described.

The user A can move the first device while touching the first device. Accordingly, the first device senses the touch input and the movement of the user A, and recognizes the device control operation corresponding to the sensed input. Also, the first device may generate and transmit the device control signal corresponding to the recognized device control operation to the second device. At this time, the second device receives the device control signal and can operate with a function corresponding to the received device control signal. That is, the second device can perform the game operation for the playing area of the user A in response to the input from the user A.

Independently, the user B can move the third device while touching the third device. Accordingly, the third device senses the touch input and movement of the user B, and recognizes the device control operation corresponding to the sensed input. Also, the third device may generate and transmit the device control signal corresponding to the recognized device control operation to the second device. At this time, the second device receives the device control signal and can operate with a function corresponding to the received device control signal. That is, the second device may perform a game operation on the playing area of the user B in response to the input from the user B.

14 to 18 are diagrams for explaining an embodiment for controlling the second device by checking the display user interface and the context information displayed on the first terminal.

14 illustrates an embodiment 1400 in which a first device is controlled to provide contextual information and an indicator user interface at a second device.

As shown in 1410, in the first device, the context information and the display user interface are displayed as icons. When the user touches the first device, the first device recognizes the device control operation corresponding to the sensed input , And generates a device control signal corresponding to the recognized device control operation.

The generated device control signal is transmitted to the second device, and in the second device, context information of the first device and an indicator user interface may be displayed as icons. In addition, the function of the second device can be performed in response to an input such as a device motion or a finger motion generated in the first device. For example, the second device can change the magnitude of the sound output in response to movement of the first device or finger movement by the user.

FIG. 15 illustrates an embodiment 1500 in which a first device is controlled to provide contextual information and an indicator user interface in a second device.

As shown in 1510, in the first device, the context information and the display user interface are presented as text, and when the user touches the first information, the first device recognizes the device control operation corresponding to the sensed input , And generates a device control signal corresponding to the recognized device control operation.

The generated device control signal is transmitted to the second device. In the second device, the context information of the first device and the display user interface may be displayed in text. In addition, the function of the second device can be performed in response to input of a device motion or a finger finger (single finger tap) generated in the first device. For example, the second device can change the thickness of the line input in response to movement of the first device or finger movement by the user.

Figure 16 illustrates an embodiment 1600 in which a first device is controlled to provide contextual information and an indicator user interface at a second device. The context information and the display user interface at this time can be presented by the color itself.

As shown in the reference numeral 1610, in the first device, the context information and the display user interface are presented as colors, and when the user touches the first device, the first device recognizes the device control operation corresponding to the sensed input And generates a device control signal corresponding to the recognized device control operation.

The generated device control signal is transmitted to the second device, and in the second device, the context information of the first device and the display user interface may be displayed as the color itself. In addition, the function of the second device can be performed in response to an input such as a device motion or a finger motion generated in the first device. For example, the second device may change the color of the line input in response to movement of the first device or two finger tap by the user.

17 illustrates an embodiment 1700 in which a first device is controlled to provide context information and an indicator user interface at a second device. At this time, context information and an indicator user interface can be presented together in a visual / audible / tactile sense.

As shown in reference numeral 1710, in the first device, the context information and the display user interface are presented in a visual / audible / tactile sense, and when the user touches the touch screen with the first device (two finger tap) Recognizes the device control operation corresponding to the input, and generates the device control signal corresponding to the recognized device control operation.

The generated device control signal is transmitted to the second device. In the second device, the context information of the first device and the display user interface may be displayed in a visual / audible / tactile sense. In addition, the function of the second device can be performed in response to an input such as a device motion or a finger motion generated in the first device. For example, the second device may change the color of the line input in response to movement of the first device or two finger tap by the user.

18 illustrates an embodiment 1800 in which a first device is controlled to provide context information and an indicator user interface at a second device. The context information and the display user interface at this time can be presented by the color itself.

As shown at 1810, the user may perform drawing and document editing through the second device. Drawing and document editing are based on the colors displayed on the first device, and the user can touch input on the screen as in 1820 to perform additional functions to change the colors displayed on the first device. When the screen is touched by the first device, the first device recognizes the device control operation (change of color) corresponding to the sensed input, and generates the device control signal corresponding to the recognized device control operation. When the user moves up and down the first device using one finger, the input color in the second device can be changed. That is, when touching and scrolling are input to the first device, an additional function through the application executable by the second device can be executed.

As shown in 1830, when a user performs a touch input using a finger to a second device and performs a gesture to raise a finger while touching the first device, the first device senses the device and can recognize the device control operation. In addition, the first device may generate a device control signal corresponding to the recognized device control action. The second device receives the generated device control signal and responds to the gesture to raise the finger, It can change and provide context information accordingly.

When the user performs a touch input using the finger to the second device and performs a gesture to lower the finger while touching the second device, the first device senses the gesture and recognizes the device control operation. The first device may generate a device control signal corresponding to the recognized device control action. The second device receives the generated device control signal and responds to the gesture of lowering the finger downward, To dark red, and context information can be provided accordingly.

FIG. 19 is a view for explaining an embodiment of detecting the change of the touch generated in the first device, changing the context information, and controlling the second device based thereon.

As shown at 1910, the change of the touch can be sensed based on the coordinate change of the touch. More specifically, assume that the X axis is the thickness of the line (0 = 0.01, xmax = 100), the Y axis is HSB space in the color space, and 0 to y0 is S (saturation) and y0 to ymax is B (brightness).

Assuming that the user's touch is Single Tap = Red (H: 0, S: 100%, B: 100%), if the touch is completed at x0, y0, If it falls, context information can be output as "red".

If the user moves with the touch at x1, y1 point at x1, y1 point, the brightness of the "red" color in the context information may change by the ratio of (y0 - y1) / (ymax - y0) . That is, the color values at points x1 and y1 have color values of (H: 0%, S: 100%, B: (y1-y0) / (ymax-y0) * 100%).

If the thickness value at x0, y0 point is w, and the thickness value at present is w, the thickness is w * (x1-x0) / (xmax-x0) when the touch point is moved to x1, Ratio.

According to this change, the visual / auditory / tactile context can also be changed and output through the output unit of the first device and the second device. In a multi-touch situation where two or more fingers are touched, the coordinates of x0 and y0 can be derived as an average of all touch points. The thickness values and color values described above are only one example and can be changed by the developer to various values.

0라고 했을 때, 사용자가 기기의 오리엔테이션(orientation)을

1으로 회전시킨 경우 최초 상태에서 임의의 볼륨(V0)는 [수학식 1]의 비율만큼 볼륨이 증가할 수 있다.Reference numeral 1920 denotes the orientation of the first device when the rotation angle of the gyro sensor is Y-axis data, assuming that the volume size of the video player (0 degree = 0, 180 degrees = 100)

0, the user can change the orientation of the device.

1, an arbitrary volume V0 in the initial state may be increased in volume by the ratio of (Equation 1).

[Equation 1]

1-

0)/(180-

0)Ratio = (

One-

0) / (180-

0)

1 상태에서의 볼륨(V1)은 아래 [수학식 2]로 계산될 수 있다.therefore,

The volume V1 in the 1 state can be calculated by the following equation (2).

&Quot; (2) "

1-

0)/(180-

0)Volume (V1) = volume (V0) * (

One-

0) / (180-

0)

At this time, the rotation by the gyro of all the devices is not limited to the y-axis data of the present example, and the same applies to the x-axis and z-axis data. In addition, the set value to be changed can be defined by various developer-defined setting values in addition to the volume.

20 is a view for explaining an embodiment for providing at least one feedback from visual feedback, tactile feedback, and auditory feedback in a second device in accordance with an operation performed by the first device;

Looking at the embodiment 2000 of FIG. 20, a visually "structured indicator UI" may be provided to a second device (e.g., iPad) to correspond to a first device (e.g., a smartphone). At this time, if a specific sensor (e.g., a gyro sensor) of the first device senses the movement of the user, the second device can change the user's structured indicator UI. For example, if the input is a 'brush' at the current location of the first device, such as 2010, the second device may express the 'brush' via the structured indicator UI. In addition, when the first device is rotated counterclockwise or clockwise as indicated by reference numeral 2020, the gyro sensor recognizes the movement of the first device, and the second device responds to the movement of the first device, Can be changed from 'brush' input to 'pencil' input.

The gyro sensor senses the movement of the first device when the movement of the first device occurs, and the second device responds to the movement of the first device that senses the movement of the first device, To 'ballpoint pen' input.

When the sensor of the first device senses the user's motion, the second device provides tactile feedback (e.g., vibration), auditory feedback (e.g., sound), etc., in addition to visual feedback to change the structured indicator UI, can do.

21 is a view for explaining another embodiment for providing at least one feedback from visual feedback, tactile feedback, and auditory feedback in a second device in accordance with an operation performed by the first device.

As shown at reference numeral 2110, when red is displayed at the current position of the first device, the second device may also be displayed to highlight red in the structured indicator UI so as to correspond to the current position of the first device.

That is, when performing a rotatable operation that can be detected by the user's gyro sensor of the first device, the color conversion can be performed in the second device instead of the conversion of the tool. At this time, the rotation of the user may be switched to the "relative rotation angle" rather than the absolute rotation angle of the first device. For example, if the user is already using the "red" pen as at 2110 and the first device is pointing at a vertical (0 degree) as at 2110, You can print the UI with the red color you are using selected.

At 2120, when the first device rotates at a predetermined angle from the current position, the first device can change the color displayed on the first device in consideration of the amount of rotation sensed by the sensor. In addition, the second device can display the changed color to be highlighted in the structured indicator UI in accordance with the changed color.

That is, if the user has performed the rotation by x degrees, a dark red rotated by x degrees from the red in the indicator UI can be selected.

Referring to reference numeral 2130, when the first device is rotated in the direction opposite to the direction of the reference numeral 2120, the color is switched to orange according to the change amount of the relative angle, and the corresponding color can be output through the indicator UI.

The motions at reference numerals 2110 to 2130 are relative to each other, and the color may be changed to correspond to the rotated angle in the case of rotating at a relatively constant angle at the previous current position.

22 is a view for explaining an operation method of the first device according to the embodiment.

An operation method of the first device according to an exemplary embodiment senses an input from a user (step 2210).

Next, the operation method of the first device recognizes the device control operation corresponding to the sensed input (step 2220), and generates a device control signal corresponding to the recognized device control operation (step 2230).

The second device transmits the generated device control signal to the second device (step 2240). The second device receives the device control signal and performs a function corresponding to the received device control signal Can operate. At this time, the second device receives the device control signal, and operates in response to the received device control signal, but can provide the function of the structured user interface based on the device control signal with the structured user interface deactivated .

The second device operates with a function corresponding to the received device control signal, and may change the previous setting value to a setting value corresponding to the device control signal. In addition, the second device may operate a function corresponding to the received device control signal, and may drive a specific additional function corresponding to the device control signal among additional functions associated with the application running on the second device.

The operation method of the first device may include receiving a second device control signal corresponding to the second device control operation collected by the second device in the data communication unit and outputting an additional function corresponding to the second device control signal received through the operation recognition unit Can be recognized. In addition, the control unit can drive the recognized additional function.

The operation method of the first device may control to provide the content generated by the driven additional function to the second device through the data communication unit to drive the recognized additional function.

On the other hand, the operation method of the first device can control to receive and output the feedback corresponding to the device control signal from the second device to drive the recognized additional function.

For example, to sense user input, at least one of pressure, frequency, speed, acceleration, direction, rotation, sound, and number of user inputs may be sensed. In order to recognize the device control operation corresponding to the sensed input, the device control operation may be recognized according to at least one of the sensed pressure, frequency, speed, acceleration, direction, rotation, sound, and number.

As a result, by using the present invention, it is possible to reduce a process of searching for a desired function by reducing unnecessary time for a user. In addition, it is possible to reduce the process of selecting functions such as touching the user unnecessarily several times, to increase the efficiency of task execution by the user, and to improve the user experience (UX) through quick switching of functions.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

210: first device 211: sensor input unit
212: Touch screen input unit 213: Operation recognition unit
214: control unit 215: output unit
216: data communication unit 220: second device
221: data communication section 222: comparison operation section
223: User setting storage unit 224:
225: control unit 226: output unit

Claims (20)

Sensing a user input through an unstructured user interface area, recognizing a device control operation corresponding to the sensed user input, generating a device control signal corresponding to the recognized device control operation, A first device for delivering the signal to a second device providing a structured user interface; And
A second device that receives the device control signal and operates in response to the received device control signal, wherein the second device provides the function of the structured user interface based on the device control signal in a state in which the structured user interface is inactive; Lt; / RTI >
Wherein the second device operates with a function corresponding to the received device control signal and changes a previous setting value to a setting value corresponding to the device control signal and when the device control signal is not received, Changes the set value to the previous set value,
Wherein the first device receives a second device control signal corresponding to a second device control operation collected by the second device and drives an additional function corresponding to the received second device control signal, And provides the content generated by the additional function to the second device. The method according to claim 1,
Wherein the second device comprises a structured user interface area comprising at least one or more indicator user interfaces wherein at least one of the unstructured user interface area and the structured user interface area comprises at least one of an icon, And outputting context information in at least one of the following forms. delete The method according to claim 1,
The second device operates a specific additional function corresponding to the device control signal among the additional functions related to the application driven by the second device while operating with a function corresponding to the received device control signal, . delete delete delete The method according to claim 1,
Wherein the first device controls to receive and output feedback corresponding to the device control signal from the second device. The method according to claim 1,
Wherein the first device senses at least one of a pressure, a frequency, a speed, an acceleration, a direction, a rotation, a sound, and a number of the user input and outputs the sensed pressure, frequency, speed, acceleration, , And the number of devices. A data communication unit for receiving a device control signal transmitted from the first device; And
A control unit operable in response to the received device control signal to control the application to provide the function of the structured user interface based on the device control signal in a state in which the structured user interface is inactive;
Lt; / RTI >
The first device senses a user input through an unstructured user interface area, recognizes a device control operation corresponding to the sensed user input, generates a device control signal corresponding to the recognized device control operation Transmits the generated device control signal to the data communication unit,
Wherein the control unit changes the previous setting value to a setting value corresponding to the device control signal while operating with a function corresponding to the received device control signal and when the device control signal is not received, To the previous setting value,
Wherein the data communication unit transmits a second device control signal corresponding to a second device control operation collected by the second device to the first device, and transmits the second device control signal corresponding to the added second device control signal from the first device, And the second device receives the content generated by the function. delete 11. The method of claim 10,
Wherein the control unit controls the application and drives a specific additional function corresponding to the device control signal among the additional functions related to the application currently being operated. 13. The method of claim 12,
And the control unit outputs the received content through the currently-running application. Sensing, at the first device, user input through an unstructured user interface area;
Recognizing a device control operation corresponding to the sensed user input in the first device and generating a device control signal corresponding to the recognized device control operation;
Transmitting, at the first device, the generated device control signal to a second device providing a structured user interface;
Receiving a device control signal from the second device and operating in response to the received device control signal, wherein the functioning of the structured user interface is disabled based on the device control signal when the structured user interface is inactive ;
Receiving, at the first device, a second device control signal corresponding to a second device control operation collected by the second device from the second device; And
In the first device, driving an additional function corresponding to the received second device control signal, and providing the content generated by the driven additional function to the second device,
Wherein the step of operating in response to the received device control signal, wherein providing the function of the structured user interface based on the device control signal comprises:
Changing a previous setting value to a setting value corresponding to the device control signal, the method comprising: changing a previous setting value to a setting value corresponding to the device control signal; And
If the device control signal is not received, changing the changed setting value to the previous setting value
≪ / RTI > delete 15. The method of claim 14,
Wherein the second device operates a function corresponding to the received device control signal and drives a specific additional function corresponding to the device control signal among additional functions related to the application driven by the second device Lt; / RTI > delete delete 15. The method of claim 14,
Wherein the step of driving the additional function comprises:
And receiving and outputting feedback corresponding to the device control signal from the second device
≪ / RTI > 15. The method of claim 14,
Wherein sensing the user input comprises:
Sensing at least one of pressure, frequency, speed, acceleration, direction, rotation, sound, and number of user inputs
Lt; / RTI >
The step of recognizing the device control operation corresponding to the sensed user input may include recognizing the device control operation according to at least one of the sensed pressure, frequency, speed, acceleration, direction, rotation, sound,
≪ / RTI >

Images