KR102046181B1 - System and method for interworking and controlling devices - Google Patents

Abstract

The present invention discloses a system for interworking and controlling terminals. The user terminal transmits information for transmitting a decoding unit for decoding image data received from a receiving terminal, a touch display unit for displaying the decoded image, and selection information about an event execution object included in the decoded image to the receiving terminal. Contains wealth.

Description

Terminal interlocking and control system and method {SYSTEM AND METHOD FOR INTERWORKING AND CONTROLLING DEVICES}

The present invention relates to a system and method for interworking and controlling terminals.

Recently, smart TVs are popular, and methods such as improving the convenience of operation in the smart TV itself have been proposed, such as Korean Patent Laid-Open Publication No. 2011-0078656. However, it is inconvenient to control the user by touching the smart TV and the remote controller for controlling the smart TV has not been effective.

The present invention provides a system and method capable of efficiently controlling a receiving terminal such as a smart TV with a user terminal such as a smart phone.

In order to achieve the above object, a user terminal according to an embodiment of the present invention includes a decoding unit for decoding the image data received from the receiving terminal; A touch display unit to display the decoded image; And an information transmitter for transmitting selection information on an event execution object included in the decoded image to the receiving terminal.

According to another embodiment of the present invention, a receiving terminal includes: an information transmitting unit transmitting all or part of an image to a user terminal; And an information receiver configured to receive selection information on an event execution object included in the image from the user terminal.

Terminal interworking method according to an embodiment of the present invention comprises the steps of sharing an image with the receiving terminal; Receiving selection information on an event execution object included in the image; And transmitting the selection information to the receiving terminal.

Terminal interworking method according to another embodiment of the present invention comprises the steps of transmitting all or part of the image to the user terminal; And receiving selection information on an event execution object included in the image from the user terminal.

A program of instructions that can be executed to interwork a user terminal and a receiving terminal according to an embodiment of the present invention is tangibly implemented, and a recording medium that can be read by a digital processing apparatus shares an image with the receiving terminal. step; Receiving selection information on an event execution object included in the image; And transmitting the selection information to the receiving terminal.

According to another embodiment of the present invention, a program of instructions that can be executed to interwork a user terminal and a receiving terminal is tangibly embodied, and a recording medium that can be read by a digital processing apparatus is the whole or all of an image to the user terminal. Transmitting a portion; And receiving selection information on an event execution object included in the image from the user terminal.

In the terminal interworking and control method and system according to the present invention, the receiving terminal provides all or part of the displayed image to the user terminal, the user can select the event execution object included in the image through the user terminal, the user The terminal transmits selection information of an event execution entity to the receiving terminal. Therefore, the user can efficiently control the operation of the receiving terminal by using the user terminal.

Since the image of the receiver terminal is displayed on the user terminal, the user can control the receiver terminal while not looking at the receiver terminal but only the user terminal. Therefore, the place restriction for controlling the receiving terminal can be eliminated.

In addition, the user terminal may be displayed a position image indicating the position of the touch means for touching near or weak pressure or area, so that the user can conveniently control the operation of the user terminal.

In addition, the location image can also be displayed on the receiving terminal, in which case the user can easily control the operation of the receiving terminal by only looking at the receiving terminal without looking at the user terminal.

1 is a view schematically showing a terminal interworking and control system according to an embodiment of the present invention.
2 is a diagram illustrating a terminal interworking and control method according to a first embodiment of the present invention.
3 is a diagram illustrating a terminal interworking and control method according to a second embodiment of the present invention.
4 is a diagram illustrating a terminal interworking and control method according to a third embodiment of the present invention.
5 is a diagram illustrating a terminal interworking and control method according to a fourth embodiment of the present invention.
6 is a diagram illustrating a terminal interworking and control method according to a fifth embodiment of the present invention.
FIG. 7 is a diagram illustrating a control method used in the terminal interworking and control method of FIG. 6.
8 is a diagram illustrating an interlocking operation when setting a sensing level according to an embodiment of the present invention.
9 is a flowchart illustrating a terminal interworking and control method according to a sixth embodiment of the present invention.
10 is a flowchart illustrating a terminal interworking and control method according to a seventh embodiment of the present invention.
11 is a diagram illustrating a terminal interworking and control system according to another embodiment of the present invention.
12 is a flowchart illustrating a terminal interworking and control method according to an eighth embodiment of the present invention.
FIG. 13 illustrates a method of detecting touch means according to a first embodiment of the present invention.
14 is a diagram illustrating a method of detecting touch means according to a second embodiment of the present invention.
15 is a diagram illustrating a method of detecting touch means according to a third embodiment of the present invention.
16 illustrates a method of detecting touch means according to a fourth embodiment of the present invention.
17 is a block diagram showing the structure of a user terminal according to an embodiment of the present invention.
18 is a block diagram showing the structure of a user terminal according to another embodiment of the present invention.
19 is a block diagram showing the structure of a receiving terminal according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Terminal interworking and control system of the present invention relates to a system for interlocking and controlling the terminals, in particular a system for controlling the operation of the terminal based on the image sharing and the shared image. In particular, the terminal interworking and control system of the present invention is a small size terminal (for example, a smartphone, a tablet PC, etc.) and a large size terminal (eg For example, various methods of controlling a large terminal by using a small terminal after interworking with a TV, etc. are proposed. That is, the terminal interworking and control system of the present invention can control a large terminal by utilizing a small terminal as a remote controller.

Hereinafter, for convenience of description, a small sized terminal directly controlled by a user is called a user terminal or a mobile terminal, and a large sized terminal that receives location information of the touch means from the user terminal will be called a receiving terminal or a display device. . Of course, the user terminal preferably has a smaller size than the receiving terminal, but the size of the terminals is not limited to the above size.

The terminal used in the system of the present invention has a function of displaying an image and a wired / wireless communication function, but the communication function is not an essential function. However, considering the real life application, it is preferable that each terminal has an image display function and a communication function. The terminal is not limited to a specific device as long as it can display an image and transmit / receive a signal with another terminal, for example, a smartphone, a smart TV, a remote controller, a PC, a tablet PC, a laptop, a touch pad, a game console, and a cloud. Various devices such as a PC can be used as the terminal of the present invention. However, it is preferable that a terminal having a small size has a touch function.

Hereinafter, a terminal interworking and control system according to various embodiments of the present invention will be described with reference to the accompanying drawings. However, for convenience of description, an image shared by the terminals will be referred to as a first image, and an image representing position information of the touch means will be referred to as a second image (position image).

1 is a view schematically showing a terminal interworking and control system according to an embodiment of the present invention.

Referring to FIG. 1A, the terminal interworking and control system of the present embodiment may include a user terminal 100 and a receiving terminal 102, and the terminals 100 and 102 may be computing devices.

The user terminal 100 may be a terminal directly controlled by a user, and may be, for example, a smartphone, a remote controller, or the like capable of sharing an image with other devices. In addition, the user terminal 100 may be a terminal having a touch function and having a relatively small size, for example, may be a mobile terminal.

The receiving terminal 102 is a terminal which is not directly manipulated by the user but interoperates with the user terminal 100, and may be a displayable terminal. That is, the receiving terminal 102 includes all devices capable of displaying images, and may be referred to as a display device in view of displaying images. The reception terminal 102 may be an element used for a different purpose than the user terminal 100, and may be, for example, a TV that broadcasts a broadcast program such as a drama. According to one embodiment, the receiving terminal 102 is a terminal having a relatively large size, but does not necessarily have a touch function.

The overall size of the receiver terminal 102 may be larger than the overall size of the user terminal 100, but it is sufficient if the size of the display unit of the receiver terminal 102 is larger than the size of the display unit of the user terminal 100. In the latter case, the overall size of the user terminal 100 and the receiving terminal 102 is not considered.

Terminals 100 and 102 may be connected directly or indirectly via another device via wired or wireless. According to an embodiment of the present disclosure, since the user controls the operation of the reception terminal 102, in detail, the operation of a program displayed on the reception terminal 102 through the user terminal 100, the user terminal 100 may be configured as a reception terminal ( It may be located at a distance capable of short-range communication with the 102, for example, a distance capable of viewing the receiving terminal 102. Of course, the communication between the terminals 100 and 102 is not limited to short-range communication, for example, the user may control the operation of the receiving terminal 102 inside the home using the user terminal 100 outside the home. Can be. This is possible because the user terminal 100 shares at least a portion of the first image displayed on the receiving terminal 102. As a result, local constraints for controlling the receiving terminal 102 can be eliminated.

According to an embodiment of the present disclosure, the receiving terminal 102 may transmit image data corresponding to at least a portion of the first image 110 displayed by the receiving terminal 102 to the user terminal 100. The user terminal 100 displays the first image 110 corresponding to the transmitted image data. That is, the user terminal 100 and the receiving terminal 102 may share the first image 110 as shown in FIG. 1B.

However, the first image 110 displayed on the user terminal 100 is substantially the same as the first image 110 displayed on the receiving terminal 102, but due to the characteristics of the user terminal 100, the color is different or The display ratio may be different. The user terminal 100 may display the transmitted image data as it is without first processing the first image 110, but after converting the resolution of the transmitted image data, etc., the first image corresponding to the converted image data 110 may be displayed. As another example, another device connected to the user terminal 100 properly converts the image data transmitted from the receiving terminal 102 to match the format of the user terminal 100 and then converts the converted image data to the user terminal 100. You can also send.

That is, as long as the receiving terminal 102 and the user terminal 100 display substantially the same first image 110, the image data processing method in the user terminal 100 may be variously modified.

According to another exemplary embodiment of the present invention, the user terminal 100 may share the first image 110 by transmitting image data corresponding to the shared first image 110 to the receiving terminal 102.

When the first image 110 on the receiving terminal 102 is changed, the receiving terminal 102 may transmit image data corresponding to the variable first image 110 to the user terminal 100. As a result, the user terminal 100 and the receiving terminal 102 may continuously share the first image 110.

In addition, when the user selects, for example, a specific event execution object among the first images 110 displayed on the user terminal 100, the user terminal 100 may provide selection information (selection information) for the event execution object. It may transmit to the receiving terminal 102. In this case, the receiving terminal 102 executes a corresponding operation according to the transmitted information, and as a result, the first image 110 may be changed. Of course, image data corresponding to the variable first image 110 is transmitted to the user terminal 100. That is, the user may control the operation of the receiving terminal 102, in particular, the operation of the program executed by the receiving terminal 102 using the user terminal 100.

The selection information may include location information corresponding to a touch input for selecting an event execution object. Alternatively, the selection information may include control information for executing the event execution object selected according to the touch input. The user terminal 100 may generate location information or control information according to a touch input.

The receiving terminal 102 may execute an operation corresponding to the event execution object by using location information corresponding to the touch input. Alternatively, the reception terminal 102 may execute an operation corresponding to the event execution entity by using the control information.

The selection of the event execution object may be determined according to the touch pressure or the area where the touch means touches the user terminal 100, and as described later, the user may exceed the preset pressure or the preset area by the touch means. The event execution object may be selected when 100) is touched. Detailed description thereof will be described later.

Hereinafter, various embodiments in which the user can control the operation of the receiving terminal 102 using the user terminal 100 will be described in detail.

According to an embodiment of the present invention, the user terminal 100 is position information of the touch means of touching the display unit near the touch sensing area of the user terminal 100, for example, the display unit or below a predetermined pressure or area. A third image 114 (position image) indicating s may be displayed as shown in FIG. That is, since the location of the touch means is displayed on the user terminal 100, the user can conveniently control the user terminal 100. The touch means is not limited, such as a finger, a touch pen. Here, the proximity means that the touch means is located within a preset distance from the user terminal 100, and may include a case in which the touch means contacts the user terminal 100.

According to another embodiment of the present invention, when the touch means approaches the user terminal 100 or touches it below a predetermined pressure or area, the receiving terminal 102 is shown in FIGS. 1B to 1D. Likewise, a second image 112 corresponding to the position of the touch means, for example, a pointer and a shadow image, may be displayed together with the first image 110. The second image 112 may be substantially the same as the third image 114 and may have a different shape or color.

That is, the position of the touch means may be specified in the user terminal 100 and the receiving terminal 102. When the second image 112 is displayed on the receiving terminal 102, the user may control the operation of the receiving terminal 102 while only looking at the receiving terminal 102 without looking at the user terminal 100.

According to an embodiment, the second image 112 or the third image 114 corresponding to the position of the touch means may be changed according to the touch pressure or the touch area of the touch means touching the user terminal 100. . For example, when the touch means touches the user terminal 100 below a predetermined pressure or a predetermined area, a third image 114 corresponding to the position of the touch means is displayed on the user terminal 100, while If the user terminal 100 is touched beyond the set pressure or area, the third image 114 may not be displayed on the user terminal 100 or may be changed into another shape. This operation may be similarly applied to the second image 112 displayed on the receiving terminal 102.

According to another embodiment, the shape of the second image 112 or the third image 114 may vary according to the distance between the touch means and the user terminal 100. For example, the image 112 or 114 has a shadow shape when the distance between the touch means and the user terminal 100 is within the first distance, whereas the distance between the touch means and the user terminal 100 is defined as the first distance. When in the second distance less than one distance may have a finger shape.

According to the embodiment, the position of the touch means close to the user terminal 100, the position of the touch means touching the user terminal 100 below the preset touch pressure, and the user terminal 100 touches the touch area below the preset touch area. At least one of the positions of the touch means may be displayed as the second image 112 or the third image 114.

In summary, the user may conveniently control the operation of the receiving terminal 102 using the user terminal 100. When the second image 112 indicating the location information of the touch means is displayed on the receiving terminal 102, the user can look at the receiving terminal 102 without looking at the user terminal 100 and use the receiving terminal 100 using the receiving terminal 100. 102 can be controlled. In particular, the user may control the receiving terminal 102 using, for example, a smartphone, even if the user does not have a separate control means.

On the other hand, since the user can control the user terminal 100 by using a touch means, the user can not only select the event execution object to execute a specific operation but also input specific characters on the user terminal 100. And work with documents. Such operation in the user terminal 100 may be immediately reflected to the receiving terminal 100.

In addition, since the touch means may be used as a mouse of a computer to perform various operations, the user may conveniently control the receiving terminal 102. For example, the user may remotely scroll, copy, or search the articles of the portal site on the user terminal 100 to reflect them on the receiving terminal 102. Of course, since the second image 112 representing the position information of the touch means is displayed on the receiving terminal 102, the user performs a desired operation while looking at the second image 112 displayed on the large receiving terminal 102. You can. That is, the present invention displays the second image 112 on the receiving terminal 102 so that the user can conveniently control the receiving terminal 102, while the user terminal 100 is a remote controller, a mouse and a touch pen, etc. You can implement various functions that can be performed. Therefore, it is possible to conveniently control the receiving terminal 102 to perform various functions such as a smart TV by using the user terminal 100 which the user always possesses.

In the above, although the user terminal 100 transmits the image data corresponding to the first image 100 and the position information of the touch means to the receiving terminal 102 separately, the user terminal 100 determines the position of the touch means. The first image 110 displayed on the first image 110 and the location of the touch means may be transmitted to the receiving terminal 102. For example, the user terminal 100 may overlap the position image of the touch means on the first image 110, and transmit the image data in which the position image of the touch means overlaps to the receiving terminal 102.

As another example, the user terminal 100 may change an area corresponding to the position of the touch means in the first image 110 to a position image of the touch means. That is, the user terminal 100 may deform the first image 110 itself to display the location of the touch means, generate a new image, and transmit the generated image to the receiving terminal 102.

According to another exemplary embodiment of the present invention, when the user does not want to display the second image 112 during the interworking, the user may control the user terminal 100 or the receiving terminal 102 by selecting a menu or operating a button. The display of the second image 112 or the third image 114 may be stopped. In this case, the user terminal 100 may continuously transmit the location information of the touch means to the receiving terminal 102 and the receiving terminal 102 may store the transmitted location information. ) May display the second image 112 corresponding to the location information.

According to another embodiment of the present invention, the linkage and control system may include a user terminal, a receiver terminal, and a linkage server. The companion server may transmit image data corresponding to the first image to the user terminal and the receiving terminal, wherein the user terminal transmits the position information of the touch means to the receiving terminal to indicate the position information of the touch means together with the first image. 2 images can be displayed on the receiving terminal. The companion server may store resolution information of the user terminal and the reception terminal in advance, process image data based on the stored resolution information, and transmit the image data to the user terminal and the reception terminal.

On the other hand, terminals such as smart phones and TVs are wired communication standards such as high-definition multimedia interface (HDMI), mobile high definition link (MHL), displayport, wireless communication standards such as DNLA and WiFi, and digital living network alliance (DNLA). They are each equipped with a separate control channel for transmitting and receiving control signals as well as a data channel for transmitting data. For example, HDMI uses CEC (Consumer Electronic Control), DDC (Display Data Channel), Utility, SCL / SDA as control channels, MHL uses CBUS as control channels, and Displayport controls auxiliary channels. I am using it as a channel. Therefore, it is not necessary to set up separate channels to interwork with the terminals 100 and 102, and channels already existing in the terminals 100 and 102 can be used as channels for transmitting location information of the touch means of the present invention. have.

Depending on the communication scheme used in the companion system, the terminals 100 and 102 may transmit and receive data in various forms.

Hereinafter, a method of specifying the position of the touch means to the receiving terminal 102 will be described.

According to an embodiment, the location information of the touch means may be location information on the first image displayed on the user terminal 100. As a result, the position of the touch means on the first image of the user terminal 100 may be reflected in the first image 110 of the receiving terminal 102 as the second image 112.

According to another embodiment, the location information of the touch means may be coordinate information according to the resolution or screen size of the display unit of the user terminal 100. That is, the position information of the touch means may be coordinate information of the touch means based on the display unit of the user terminal 100, not coordinate information of the touch means on the image displayed on the screen.

According to another embodiment, when the user terminal 100 and the receiving terminal 102 wants to display image data transmitted from the receiving terminal 102 in order to share the first image 110. The user terminal 100 may set the screen to the same resolution as the receiving terminal 102 and then display the first image 110 corresponding to the image data. Therefore, when the user terminal 100 displays the location information of the touch means, for example, the second image 112 as it is, the coordinate information may accurately represent the position of the touch means.

That is, the present invention can specify the location of the touch means to the receiving terminal 102 through various methods. The user terminal 100 may transmit location information of the touch means to the receiving terminal 102, generate second image data corresponding to the second image indicating the location information, and transmit the second image data to the receiving terminal 102, or touch the touch terminal. It is also possible to transmit to the receiving terminal 102 a first image of a modified area corresponding to the position of the means.

Hereinafter, various embodiments of the terminal interworking and control method of the present invention will be described with reference to the accompanying drawings.

2 is a diagram illustrating a terminal interworking and control method according to a first embodiment of the present invention.

Referring to FIG. 2A, the reception terminal 102 may receive image data corresponding to the first image 110 including the event execution object 200 such as, for example, a UI, an icon, an execution program, a link, and the like. It may transmit to the user terminal 100. As a result, the event execution object 200 may be included in the first image 110 displayed on the user terminal 100.

According to an embodiment of the present disclosure, when the user touches the event execution object 200 by the touch means, the user terminal 100 may transmit selection information indicating that the event execution object 200 is selected to the reception terminal 102. The selection information may include location information corresponding to a touch input for selecting the event execution object 200 or control information for executing the event execution object 200 according to the touch input.

The receiving terminal 102 may determine that the event execution entity 200 is selected through the information corresponding to the touch input, and execute an operation related to the selection of the event execution entity 200. Alternatively, the receiving terminal 102 may execute a related operation by using control information for executing the event execution object 200. Therefore, the user may control the operation of the receiving terminal 102 using the user terminal 100.

At this time, when a user approaches the touch means to the event execution object 200 of the user terminal 100 or touches it under a predetermined pressure or area, the receiving terminal 102 is provided at a position corresponding to the position of the touch means. 2 images 112 may be displayed. Of course, the third image 114 indicating the position of the touch means may also be displayed on the user terminal 100. Subsequently, when the user touches the event execution object 200 by a touch means with a pressure larger than a predetermined pressure or an area larger than a predetermined area, the receiving terminal 102 performs an operation corresponding to the event execution object 200. For example, a game may be executed. In this case, the first image 110 displayed on the receiving terminal 102 may be changed.

The terminal interworking method can be very useful when controlling not only games, navigation, etc., but also the smart TV as shown in FIG. 2B using the user terminal 100.

3 is a diagram illustrating a terminal interworking and control method according to a second embodiment of the present invention.

Referring to FIG. 3, only the image 300 corresponding to the UI for control among the first images 110 displayed on the receiving terminal 102 is transmitted to the user terminal 100, and the corresponding image is displayed on the user terminal 100. 302 may be displayed. That is, the user terminal 100 and the receiving terminal 102 may share only a part of an image, particularly a part including an event execution object. In this case, the receiving terminal 102 displays a second image 112 indicating the position of the touch means. Of course, the third image 114 indicating the position of the touch means may also be displayed on the user terminal 100, and the third image 114 may be substantially the same as the second image 112.

In such a coordinated system, when a user touches an event execution object among the images 302 displayed on the user terminal 100 with a touch means, the user terminal 100 transmits selection information of the event execution object to the receiving terminal 102. The transmitting and receiving terminal 102 may execute an operation corresponding to the selection information. In general, the first image 110 on the receiving terminal 102 will change. In this case, if the image 300 of the event execution object of the receiving terminal 102 does not change, the user terminal 100 maintains the image 302, but the image 300 of the event execution object of the receiving terminal 102 If changed, the user terminal 100 displays an image 302 of the changed event execution object.

On the other hand, when the touch means is close to the display of the user terminal 100, the second image 112 or the third image 114 is displayed. In particular, when the touch means is positioned on the event execution object, event occurrence information such as sound is output. You can also This is described later in detail in FIG. 18.

In summary, the present invention may use the user terminal 100 as a means for controlling the receiving terminal 102. In particular, this approach can be effective in games, e-commerce, smart TVs, and the like.

4 is a diagram illustrating a terminal interworking and control method according to a third embodiment of the present invention.

Referring to FIG. 4, the image of the second image 112a indicating the position of the touch means may be modified into another shape. In addition, the second image 112a may be changed when the touch means is present at a preset position in a state in which the touch means is close to the user terminal 100 or in a state where the user terminal 100 is touched at a preset pressure level or area or less. .

For example, the second image 112 is represented with a shadow image as shown in FIG. 4A when the touch means is not positioned on the event execution object, but the event execution object of the touch means, for example When positioned on the execution icon, it may be represented with a finger image as shown in FIG. In addition to the icon, the second image 112 may be changed even when the touch means is located in an internet address input window, a bottom of the screen, a search window, a folder, and the like.

In this case, the receiving terminal 102 may output event occurrence information such as sound, light, vibration, etc. according to an event where the touch means is positioned on an event execution object. Detailed description thereof will be described later. The deformation of the second image 112 and the output of the event occurrence information may be simultaneously performed. The above method can be similarly applied to the third image 114.

According to another embodiment, the image 112 or 114 may be changed according to the number of times the touch means touches the user terminal 100 or the touch time. For example, the image 112 or 114 may have a shadow image when the touch means touches the user terminal 100 once, and may be changed to an arrow image when touched twice in succession. As another example, the image 112 or 114 may have a shadow image when the touch means touches the user terminal 100 for less than a preset time, and may be changed to another image when the touch means exceeds the preset time.

As another example, the image 112 or 114 when the touch means approaches the user terminal 100 may be different from the image 112 or 114 when the touch means touches the user terminal 100.

Meanwhile, although the second image 112 and the third image 114 may be changed at the same time, only one or a different image may be changed.

Hereinafter, the operation of the terminal interworking and control system of the present invention will be described in detail with reference to the accompanying drawings.

5 is a diagram illustrating a terminal interworking and control method according to a fourth embodiment of the present invention.

Referring to FIG. 5, the user terminal 100 and the receiving terminal 102 are connected to start interlocking (S500). Specifically, when the user is connected to the user terminal 100 and the receiving terminal 102 for interworking, the user terminal 100 or the receiving terminal 102 requests the other terminal to interwork, or the user terminal 100 If the interworking request is made to the receiving terminal 102 and the interworking is accepted, a channel for transmitting the location information of the touch means may be formed between the terminals 100 and 102. Here, the interworking of the terminals 100 and 102 may be performed at the request of the user, or may be automatically performed by the terminal 100 or 102.

According to an embodiment, the user terminal 100 and the receiving terminal 102 may be connected to two channels, that is, a data channel and a control channel, and may transmit and receive a control signal for interworking request or acceptance through the control channel. have.

Subsequently, the receiving terminal 102 transmits image data corresponding to at least a portion of the first image currently displayed or at least a portion of the first image to be displayed to the user terminal 100, and the user terminal 100 transmits the transmitted image data. The first image corresponding to the image data is displayed (S502). That is, the user terminal 100 and the receiving terminal 102 share at least a portion of the first image.

Subsequently, when a user moves a touch means such as a finger or a touch pen to the user terminal 100 or touches the user terminal 100 with a predetermined pressure or area or less, the user terminal 100 detects the touch means. (S504). The user terminal 100 may detect the touch means by using various methods such as a capacitance sensing method and an electromagnetic sensing method.

Subsequently, the user terminal 100 transmits the location information of the touch means to the receiving terminal 102 according to the detection result, and the receiving terminal 102 first transmits a second image indicating the position of the transmitted touch means. The display is displayed with the image (S506).

According to another exemplary embodiment of the present invention, the user terminal 100 transmits image data (or composite image data) including data corresponding to the first image and data corresponding to the second image, to the receiving terminal 102. It may be. However, since the reception terminal 102 basically transmits the image data corresponding to the first image to the user terminal 100, the user terminal 100 transmits only the location information of the touch means to the reception terminal 102. The former method is efficient.

Subsequently, when the user touches or moves the touch means, the user terminal 100 transmits the position information of the touch means to the receiving terminal 102 to display the second image on the receiving terminal 102 (S508). That is, the movement of the touch means may be reflected on the receiving terminal 102.

6 is a diagram illustrating a terminal interworking and control method according to a fifth embodiment of the present invention, FIG. 7 is a diagram illustrating a control method used in the terminal interworking and control method of FIG. 6, and FIG. 8 is a diagram of the present invention. 1 is a diagram illustrating an interlocking operation when setting another sensing level.

Referring to FIG. 6, the user terminal 100 or the receiving terminal 102 requests interlocking to start interlocking (S600).

According to an embodiment, as shown in FIG. 7A, the user terminal 100 and the receiving terminal 102 may be connected to two channels, that is, a data channel and a control channel, and may request or accept an interworking request. The control signal can be transmitted and received through the control channel.

According to another embodiment, data may be transmitted and received through one channel. For example, as shown in (C) of FIG. 7, the channel transmission section is a control section that transmits position information or selection information of an event execution object between data sections E1 and E2 for transmitting image data. (C) may be included. In particular, the location information or the selection information may be transmitted by using a blank period (C) between the sections for transmitting the image data. For example, the user terminal 100 transmits image data to the receiving terminal 102 through one channel, and during the blank period C existing between the data sections E1 and E2 for transmitting the image data. The location information or selection information may be transmitted to the receiving terminal 102.

Subsequently, the reception terminal 102 transmits image data corresponding to at least a portion of the first image 110 to the user terminal 100 to share the first image 110 with the user terminal 100 (S602).

Subsequently, the user terminal 100 detects a touch pressure or a touch area of the touch means (S604).

For example, in order to sense the touch pressure or the touch area, the user terminal 100 may set the sensing level to a plurality of levels, for example, two levels, as shown in FIG. 7B. When not performing the terminal interworking process, that is, when the user uses only the user terminal 100, the detection level of the user terminal 100 is set to a high level (L2) so that the touch position is detected only when the user touches weakly. Can be.

On the other hand, when the user performs the terminal interworking process, that is, also using the receiving terminal 102, the user terminal 100 changes the detection level to a low level (L1). Therefore, the user terminal 100 may detect the touch means even when the touch means is not in contact with the touch means or when the touch pressure or the touch area is equal to or less than the preset pressure or area. According to another embodiment of the present invention, the user terminal 100 may use a variety of methods, such as electromagnetic induction, resistive film, optical, ultrasonic method, in addition to the capacitive method, the setting conditions may vary depending on the usage method .

Subsequently, the user terminal 100 transmits the position information of the touch means according to the detection result to the receiving terminal 102, and the receiving terminal 102 shows a second image representing the position information, for example, FIG. 1B. As shown in FIG. 6, the position image 112 indicating the position of the touch unit is displayed together with the first image 110 (S606).

In this case, the position information of the touch means may be information in the form of an image or information in the form of a coordinate. That is, the user terminal 100 directly generates the position information of the second image 112 in the form of image data and transmits it to the receiving terminal 102 or only the position information of the touch means in the form of a control signal. Can be sent to. Alternatively, the user terminal 100 shares the first image 110 by generating a location image (second image) of the touch means and transmitting image data including the first image and the second image to the receiving terminal 102. At the same time, the second image 112 may be displayed on the receiving terminal 102.

For example, when the touch pressure of the touch means is equal to or less than the preset pressure or the touch area is equal to or less than the preset touch area, the position of the touch means may be displayed on the receiving terminal 102. That is, when the touch means weakly touches the user terminal 100, the second image 112 may be displayed on the receiving terminal 102. When the touch means moves while the user terminal 100 is weakly touched, the second image 112 reflects the movement of the touch means, so that the second image 112 on the receiving terminal 102 also moves continuously. (S608).

According to the present invention, when the touch means weakly touches the user terminal 100, the user terminal 100 does not recognize the touch of the touch means as a touch input. When the touch means strongly touches the user terminal 100, that is, when the touch pressure exceeds the preset pressure or the touch area exceeds the preset touch area, the user terminal 100 uses the touch means as a touch input. Recognize. Therefore, when the touch means weakly touches the icon on the icon on the user terminal 100, the icon is not executed. However, when the touch is strongly touched, the icon may be executed.

As described above, the embodiment of FIG. 5 and the embodiment of FIG. 6 may be applied together. That is, when the touch means is located within a preset distance from the user terminal 100 and when the touch means weakly touches the user terminal 100, the second image 112 may be displayed on the receiving terminal 102. have.

The detection level setting shown in the embodiments of FIGS. 5 and 6 will be described in more detail with reference to FIG. 8.

A plurality of sensing levels may be set in the user terminal 100. For example, the user terminal 100 includes a first level for detecting proximity of the touch means 804 and a second for detecting the touch means 804 for touching below a predetermined level (pressure level or area level). A third level for detecting the level and the touch means 804 for touching above the predetermined level may be set.

The operation of the user terminal 100 and the receiving terminal 102 when these multiple levels are set will be described.

The receiving terminal 102 may display the first image 110 as shown in FIG. 8A. Of course, a substantially identical first image 110 may also be displayed on the user terminal 100. The first image 110 may include, for example, an event execution object 800 such as an icon, an execution program, a link, and the like. For example, Angry Birds games can be played. Hereinafter, it will be assumed that the touch position of the touch means 804 corresponds to a specific point on the event execution object 800.

First, as shown in FIG. 8C, when the touch means 804 is close to the display unit 806 of the user terminal 100, the receiving terminal 102 is shown in FIG. 8A. As shown, a second image 112 representing the position of the touch means 804 may be displayed. In addition, the third image 114 may be displayed on the user terminal 100.

Subsequently, when the touch means 804 touches the display unit 806 below a predetermined level as shown in FIG. 8D, the event execution object 800 is not executed and the touch means 804 of the touch means 804 does not execute. An image 112 or 114 representing the position may be maintained.

Subsequently, when the touch means 804 touches the display unit 806 above a predetermined level as shown in FIG. 8E, the user terminal recognizes the selection as the event execution object 804 to play the game. You can run On the other hand, the terminal 810 of the touch means 804 may have a structure that can be inserted into the interior, and when the user touches the touch means 804 above a predetermined level as shown in (E) of FIG. As described above, the display unit 806 is pressed with the end 810 inserted therein.

In summary, the interlocking and control system of the present embodiment may perform different operations for each sensing level.

According to another embodiment of the present invention, when the touch means 804 is close, an image 112 or 114 indicating the position of the touch means 804 is displayed and the touch means 804 may have touched the display unit 806. The event execution object 800 may be executed.

9 is a flowchart illustrating a terminal interworking and control method according to a sixth embodiment of the present invention.

Referring to FIG. 9, the user terminal 100 and the receiving terminal 102 start interlocking (S900).

Subsequently, the reception terminal 102 transmits image data corresponding to at least a portion of the displayed first image to the user terminal 100, and the user terminal 102 displays the first image corresponding to the image data ( S902). That is, the user terminal 100 and the receiving terminal 102 share the first image.

Subsequently, the user terminal 100 detects touch means such as a finger, a touch pen, and the like through various methods (S904). The user terminal 100 may detect the position of the touch means close to the user terminal 100 or the touch means weakly touching the user terminal 100.

Subsequently, the user terminal 100 generates a composite image including a second image corresponding to the position of the detected touch means together with the first image currently displayed, and synthesizes image data corresponding to the composite image (synthesis information). ) Is transmitted to the receiving terminal 102, and the receiving terminal 102 displays the synthesized image corresponding to the synthesized image data (S906). As a result, the fisheries terminal 102 displays a second image together with the first image. In this case, only the first image may be displayed on the user terminal 100, or the second image may be displayed together with the first image.

Subsequently, when the user touches or moves in the state of touching the touch means, the user terminal 100 transmits the position information of the touch means to the receiving terminal 102, and the receiving terminal 102 is connected to the position information. Accordingly, the second image indicating the position of the touch means is displayed together with the corresponding first image (S908). That is, the movement of the touch means is reflected on the screen of the receiving terminal 102. However, the first image on the receiving terminal 102 may be the same image as the previous image or may be different from the previous image.

In summary, in the present embodiment, the user terminal 100 generates a composite image including the first image and the second image and transmits the generated composite image to the receiving terminal 102, and as a result, the receiving terminal 102. The second image is displayed along with the first image.

In the above, although expressed as a composite image, the first image itself may be modified such that the first image indicates the position of the touch means. Specifically, the user terminal 100 transforms an area corresponding to the position of the touch means among the first images into a shadow image, that is, transforms the first image itself to generate a new image, and receives the generated image. It may be transmitted to the terminal 102. Here, the modified first image may be substantially the same as the composite image.

10 is a flowchart illustrating a terminal interworking and control method according to a seventh embodiment of the present invention.

Referring to FIG. 10, the user terminal 100 and the receiving terminal 102 start interlocking (S1000).

Subsequently, the user or the user terminal 100 sets a plurality of sensing levels for sensing the touch means (S1002). As described above, various levels may be set according to the embodiment. This setting may be set at the start of the interworking or may be set in advance in the user terminal 100 before starting the interlocking.

Subsequently, the user terminal 100 detects the touch means, and the receiving terminal 102 displays a second image indicating the position of the detected touch means (S1004).

Subsequently, it is determined whether or not there is a request for interlocking stop by the user (S1006). The interworking stop may be requested by the user by selecting a menu or by turning off the connection between the user terminal 100 and the receiving terminal 102. When the user controls the user terminal 100 while viewing the receiving terminal 102 and wants to use only the user terminal 100 or only wants to view the receiving terminal 102, the user terminal 100 and the receiving terminal 102 You can request to stop interlocking while connected.

If there is no user stop request for interworking, step S1004 is performed again.

On the other hand, when the user stops the interlocking request of the user, the user terminal 100 initializes the plurality of levels to change to sensing levels for detecting only one level or only touch (S1008). That is, when the touch means is not touched, the user terminal 100 does not sense the touch means but changes the level so as to detect the touch means only when the touch means is touched.

In summary, in the terminal interworking and control method of the present invention, a user may freely set and change an interworking request, an interlocking stop request, and a sensing level.

According to another embodiment of the present invention, the first detection level may be set when the user terminal 100 and the reception terminal 102 start interlocking, and the detection level may be differently set during the interlocking. For example, the user may set different sensing levels according to the approach distance and the touch intensity of the touch means during interlocking, or change the sensing level setting to sense only when the touch means is touched.

11 is a diagram illustrating a terminal interworking and control system according to another embodiment of the present invention.

Referring to FIG. 11A, the terminal interworking and control system of the present embodiment may include a user terminal 100, a reception terminal 102, and a transmission / reception device 1100 (eg, a dongle).

The transceiver 1100 may connect communication between the user terminal 100 and the receiver terminal 102. Specifically, when the user terminal 100 transmits the position information of the touch means, the composite image data, or the selection information of the event execution object to the transceiver 1100, the transceiver 1100 transmits the position information of the touch means and the synthesized information. Image data or selection information may be transmitted to the receiving terminal 102. In addition, the transceiver 1100 may transmit the image data transmitted from the reception terminal 100 to the user terminal 100.

The transceiving device 1100 may be connected to the user terminal 100 or the receiving terminal 102, or may exist separately without being connected to the user terminal 100 and the receiving terminal 102. The transceiver 1100 may be, for example, a dongle, a settop, or the like.

According to an embodiment, the transceiving device 1100 may transmit the location information, the composite image data, or the selection information transmitted from the user terminal 100 to the receiving terminal 102 without being converted.

According to another exemplary embodiment, the transceiving device 1100 converts the location information, the composite image data, or the selection information transmitted from the user terminal 100 to match the format of the receiving terminal 102, and converts the converted location information or the composite image. Data may be sent to the receiving terminal 102. Since many companies currently manufacture a receiving terminal 102 such as a smart TV, it is necessary to match the format of the receiving terminal 102 with the location information, composite image data, or selection information for each manufacturer. According to the present invention, since the location information, the composite image data, or the selection information is modified to fit the format of the receiving terminal 102 by using the transmission / reception apparatus 1100, the terminals 100 and 102 may be linked regardless of the manufacturer. .

Although not shown, the transceiver 1100 may include a communication unit, a signal unit, and a format changer.

The communication unit connects the user terminal 100 and the receiving terminal 102.

The signal unit transmits the position information, the composite image data or the selection information of the touch means transmitted from the user terminal 100 to the receiving terminal 102, and transmits the first image received from the receiving terminal 102 to the user terminal 100. Can be sent to.

The format changer may change location information, composite image data, or selection information to match the format of the receiving terminal 102 or change the first image to match the format of the user terminal 100.

In summary, the user terminal 100 and the receiving terminal 102 may transmit and receive location information, image data, composite image data, or selection information through the transceiver 1100. In this case, the user terminal 100 or the receiving terminal 102 connected to the transceiver 1200 may not have a communication function.

According to another embodiment, the transceiving device 1100 may perform a role of detecting the touch means 1102 as well as a communication role between the user terminal 100 and the receiving terminal 102.

According to another embodiment, the transmission and reception apparatus 1100 may provide a specific communication function, for example, a Wibro communication function as a communication means for the user terminal 100, and may convert the specific communication function into another communication function. For example, Wibro may be converted to Wi-Fi and used for the user terminal 100.

12 is a flowchart illustrating a terminal interworking and control method according to an eighth embodiment of the present invention.

Referring to FIG. 12, a receiver is installed in the receiving terminal 102 (S1200). According to another embodiment of the invention, the receiver may be embedded in the receiving terminal 102.

Subsequently, the user terminal 100 and the receiving terminal 102 start interlocking (S1202).

Subsequently, the user terminal 100 transmits image data corresponding to the first image 110 to the receiving terminal 102 so as to share the first image 110 or the receiving terminal 102 to the user terminal 100. Image data is transmitted to share the first image 110 (S1204).

Subsequently, when the touch means approaches the user terminal 100 or touches the display unit of the user terminal 100 with a predetermined pressure or area or less, the receiver detects the position of the touch means through infrared rays and ultrasonic waves transmitted from the touch means. In operation S1206, the location of the detected touch means is transmitted to the reception terminal 102, and the reception terminal 102 displays the second image 112 indicating the detected location together with the first image 110 (S1206).

Subsequently, the user terminal 100 transmits the position information according to the movement of the touch means to the receiving terminal 102, and the receiving terminal 102 differs from the second image 112 or the second image. It can be displayed as a third image.

Hereinafter, various methods of sensing the touch means will be described with reference to the accompanying drawings.

FIG. 13 illustrates a method of detecting touch means according to a first embodiment of the present invention.

Referring to FIG. 13, the touch pen 1300 may be used as a touch means for touching the user terminal 100. The user terminal 100 uses a capacitive touch panel.

The touch pen 1300 may be formed of a body 1310 and a touch unit 1312. The body 1310 is made of a material that does not conduct electricity, and the touch unit 1312 is a conductor. Accordingly, a change in capacitance occurs when the touch pen 1300 approaches the user terminal 100 or touches the user terminal 100 due to the touch unit 1312. The touch pen 1300 may be detected through the change.

14 is a diagram illustrating a method of detecting touch means according to a second embodiment of the present invention.

Referring to FIG. 14, the user terminal 100 may include a touch panel 1400 and an electromagnetic generator 1402.

The electromagnetic generator 1402 may be connected to the rear surface of the touch panel 1400, and may be formed of a thin metal film to generate an electromagnetic field when electricity flows.

The touch pen 1404 includes a body 1410 and a touch unit 1412, and the touch unit 1412 may be preferably made of a small metal coil. As a result, when the touch pen 1404 is close to the touch panel 1400, an electromagnetic induction phenomenon occurs in the touch unit 1412, and as a result, a deformation of the electromagnetic field generated from the electromagnetic generator 1402 occurs. Therefore, the user terminal 100 detects the deformation of the electromagnetic field and recognizes the position of the touch pen 1404. In particular, since the deformation of the electromagnetic field varies depending on the approach position and the touch intensity of the touch pen 1404, the touch means sensing method detects the touch pen 1404 approaching the touch panel 1400 and the touch pressure in detail. can do.

15 is a diagram illustrating a method of detecting touch means according to a third embodiment of the present invention.

Referring to FIG. 15A, a receiver 1500 may be mounted on a part of the user terminal 100, and a touch pen 1502 may be used.

The receiver 1500 may include one infrared sensor and two ultrasonic sensors, and detects the movement of the touch pen 1502 by receiving infrared rays and ultrasonic waves transmitted from the touch unit (tip) of the touch pen 1502. In response to the detection result, location information of the touch pen 1502 is transmitted to the reception terminal 102. The receiving terminal 102 displays a second image representing the transmitted location information. As a result, the second image is displayed together with the first image. In this case, the location information may be transmitted by the receiver 1500 to the receiving terminal 102 or the user terminal 100 may be transmitted to the receiving terminal 102.

According to another embodiment of the present invention, the receiver 1500 may perform not only a function of detecting the position of the touch pen 1502 but also a function of transmitting image data and position information to the receiving terminal 102. Specifically, the receiver 1500 may include a touch controller 1510, an image signal unit 1512, a control signal unit 1514, and a transceiver 1516 as shown in FIG. 15B. .

The touch controller 1510 detects the position of the touch pen 1502 by using the received infrared rays and ultrasonic waves, and provides information on the detected position to the user terminal 100. The user terminal 100 displays the location of the touch pen 1502 or performs an associated operation according to the provided information.

The image signal unit 1512 may receive image data from the user terminal 100, and transmit the provided image data to the receiving terminal 102 through the transceiver 1516.

The control signal unit 1514 transmits a control signal having the detected position information of the touch pen 1502 to the receiving terminal 102. That is, since the receiver 1500 transmits the image data transmitted from the user terminal 100 and the position information of the touch pen 1502 to the receiving terminal 102, the user terminal 100 does not need to have a communication function. Accordingly, a terminal having no communication function or a terminal having a communication function but unable to use a related communication facility may not only recognize the position and operation of the touch pen 1502 using the receiver 1500 but also share an image and a second device. The interlocking method of the present invention, such as display of an image, can be used.

According to another embodiment of the present invention, the receiver may be included in the user terminal 100 to be integrally implemented.

16 illustrates a method of detecting touch means according to a fourth embodiment of the present invention.

Referring to FIG. 16A, in the terminal interworking and control system of the present invention, the receiver 1600 may be installed or embedded in the receiving terminal 102 instead of the user terminal 100.

When the touch pen 1602 transmits ultrasonic waves and infrared rays to the receiver 1600 installed in the receiving terminal 102 as a touch means used in the user terminal 100, the receiver 1600 receives the ultrasonic waves and infrared rays and touch pens. Detect the position of 1602.

The receiver 1600 transmits information about the detected position of the touch pen 1602 to the receiving terminal 102, and the receiving terminal 102 transmits a second image indicating the position of the touch pen 1602 to the first image. Display with. Of course, the user terminal 100 and the receiving terminal 102 display the second image in a state in which the first image is shared.

According to another embodiment of the present invention, the receiver 1600 may perform a communication role as well as position detection of the touch pen 1602. In detail, the receiver 1600 may include a touch means detector 1610, an image signal unit 1612, a control signal unit 1614, and a transceiver 1616.

The touch means detector 1610 detects the position of the touch pen 1602.

The image signal unit 1612 may receive the composite image data transmitted from the user terminal 100 through the transceiver 1616 and transmit the received image data to the receiver terminal 102 through the transceiver 1616. In addition, the image data may be received from the reception terminal 102 and the received image data may be transmitted to the user terminal 100.

The control signal unit 1614 may receive a control signal related to an interlocking operation and the like transmitted from the user terminal 100 and transmit the received control signal to the receiving terminal 102. Of course, the location information of the touch pen 1602 is not transmitted from the user terminal 100 to the receiver 1600. In conclusion, the receiver 1600 performs a communication function as well as a function of detecting the position of the touch pen 1602. Therefore, the receiving terminal 102 can use the interworking and control method of the present invention without having a communication function.

17 is a block diagram showing the structure of a user terminal according to an embodiment of the present invention.

Referring to FIG. 17, the user terminal 100 of the present embodiment includes a control unit 1700, an interlocking unit 1702, an image unit 1704, a sensing unit 1706, a setting unit 1708, a display unit 1710, It may include a signal unit 1712, a transceiver 1714, a decoder 1716, and a storage 1718.

The interlocking unit 1702 manages all functions related to interworking with the receiving terminal 102.

The image unit 1704 may include an image generator and an image changer, and may display an image corresponding to the image data transmitted from the receiver terminal 102 through the display unit 1710.

The image generator may generate a position image indicating the position of the touch means located within a predetermined distance from the display unit 1710 or generate a composite image including the second image and the first image, and display unit 1710. It may also generate a position image of the touch means touched below a predetermined pressure or area.

The image changing unit may change the position image according to the distance between the touch means and the user terminal 100, or change the position image according to the pressure or area of the touch means touching the user terminal 100. In addition, the image changing unit may change the position image depending on whether the touch means exists at the event execution object or a predetermined position.

The sensor 1706 detects a touch means such as a finger or a touch pen. Specifically, the sensing unit 1706 may detect the position of the touch means by distinguishing when the touch means is close to when it is touched. The sensing method is not limited, such as capacitive and electromagnetic induction. The information about the position of the touch means detected by the detector 1706 may be generated by the location information generator (not shown).

The setting unit 1708 manages setting of various functions such as setting of an interlocking function and setting of a detection level.

The display unit 1710 may be implemented in various ways, such as a capacitive type, a resistive film type, and an electromagnetic induction type. The display unit 1710 may be a touch display unit having a touch function.

The signal unit 1712 may include an information transmitter 1720 and an information receiver 1722.

The information transmitter 1720 may transmit location information of the touch means or selection information on the event execution object to the receiving terminal 102. The selection information may be position information corresponding to a touch input for selecting an event execution object or control information for executing the event execution object according to the touch input.

The information receiving unit 1722 receives image data corresponding to all or part of the first image from the receiving terminal 102.

The transceiver 1714 serves as a communication path with the receiving terminal 102.

The decoding unit 1716 decodes the image data received from the receiving terminal 102.

The storage unit 1718 stores various data such as a first image, an image signal, location information, a control signal, and an application program.

The controller 1700 generally controls the operations of the components of the user terminal 100.

Although not described above, the user terminal 100 may further include a touch means unit, a receiver unit, an electromagnetic generator or a touch means operation unit.

The touch means unit receives and manages information on the position of the touch means from the receiver when the receiver is connected to the user terminal 100. That is, the position of the touch means is detected by the receiver, and the touch means unit detects the position of the touch means by analyzing a signal transmitted from the receiver.

The receiver unit receives infrared rays and ultrasonic waves transmitted from the touch means when the touch means approaches or touches the user terminal 100, and analyzes the received infrared rays and ultrasonic waves to detect the position of the touch means.

The electromagnetic generator plays a role of generating electromagnetic waves in order to sense the touch means by an electromagnetic induction method, and may be formed on the rear surface of the display unit.

The touch means operating unit may perform a specific operation when the touch means approaches the user terminal 100 or touches the user terminal 100 with a predetermined pressure or area or less. For example, when the touch means approaches the lower end of the display unit 1710 of the user terminal 100, a scroll function may be performed. In this case, the location information of the touch means may be transmitted to the receiving terminal 102 or the third image 114 corresponding to the location information may be displayed on the display unit 1710. That is, the user terminal 100 transmits the position information of the touch means to the receiving terminal 102 in response to the access or weak touch of the touch means, or displays the third image 114 on the user terminal 100 such as scrolling or the like. You can perform a specific action.

18 is a block diagram showing the structure of a user terminal according to another embodiment of the present invention.

Referring to FIG. 18, the user terminal 100 of the present exemplary embodiment may include a controller 1800, a display 1802, a detector 1804, a signal unit 1806, an image unit 1808, and an information provider 1810. And a mode changer 1812. Meanwhile, the user terminal 100 may include all of the above components or only some of them. In addition, the user terminal 100 may further include other components in addition to the above components.

The display 1802 displays a first image shared by the user terminal 100 and the receiving terminal 102. In addition, the display 1802 may display a menu for selecting a touch mode.

When the touch means approaches or touches the user terminal 100, the sensing unit 1804 detects the position of the touch means through various methods as described above. Here, the position information indicating the position of the touch means may be position information of the touch means located within a predetermined distance from the user terminal 100.

The signal unit 1806 transmits the detected position information of the touch means to the receiving terminal 102 and receives image data corresponding to the first image shared by the user terminal 100 and the receiving terminal 102. From).

The image unit 1808 may generate a composite image data to be shared between the user terminal 100 and the receiving terminal 102 or a second image indicating the position of the touch means.

The information provider 1810 may output information according to a detection result of the detector 1804. That is, when the touch means is detected by the detector 1804 in proximity to the user terminal 100, proximity information may be output. Proximity information may be in the form of vibration, sound or light to stimulate the user's sense of touch, hearing or vision.

On the other hand, the information providing unit 1810 according to the state in which the touch means is close to the user terminal 100, the user of the user terminal 100 in a variety of types so that the user can feel a variety of tactile, auditory or visual sense, the sense of touch, sight or hearing Can be provided to the user.

For example, the information providing unit 1810 may generate continuous vibration, sound, or light while the touch means moves while the user terminal 100 recognizes a proximity touch of the touch means. That is, when the proximity touch of the first touch means is recognized, one short vibration may be generated, and when the touch means moves later, vibration may be continuously generated. In other words, when the proximity touch of the first touch means is recognized, vibration may occur during the first time, and when the touch means moves later, vibration may occur during the second time. The second time may be longer than the first time.

Alternatively, vibration may be provided when the first proximity touch is recognized, and sound may be provided from the information providing unit 1810 when the touch means is moved.

As another example, the information providing unit 1810 may generate proximity information such as a sound to the user when the touch means approaches a preset object such as a folder or a control UI among images displayed on the screen of the user terminal 100. That is, as illustrated in FIGS. 3 and 4, when the touch means exists at a preset position, the position image of the touch means may be changed. At this time, the information providing unit 1810 may output the proximity information and the proximity The information may correspond to event occurrence information. Thus, the user can immediately recognize the object.

The information providing unit 1810 provides the proximity information for a first time when the sensing unit 1804 recognizes the proximity state of the touch means, and the sensing unit 1804 recognizes the proximity state of the touch means. When the touch means moves in, the proximity information may be provided for a second time. The second time may be a time corresponding to a time when the touch unit moves in a state where the sensing unit 1804 recognizes a proximity state of the touch unit.

Also, the information providing unit 1810 may include a first case in which the sensing unit 1804 recognizes the proximity state of the touch means and a second in which the touch means moves in the state in which the sensing unit 1804 recognizes the proximity state of the touch means. In this case, different types of proximity information may be provided. As described above, in the first case, vibration may be provided in the second case, or in the second case, the first pattern may be vibrated. In the second case, the second pattern may be vibrated so that the user can feel the movement of the touch means. Can provide.

On the other hand, when the touch means touches the display 1802 after the sensor 1804 recognizes the proximity state, the information provider 1810 does not provide proximity information, so that the user may directly touch the proximity state. Can be distinguished.

In addition, the information providing unit 1810 may provide proximity information when the sensing unit 1804 recognizes the proximity state of the touch unit for the second time after the touch unit touches the display unit 1802. When the touch means is close to the user terminal 100 and touches the user terminal 100, the touch means is spaced apart from the user terminal 100. That is, since the touch object is achieved, the touch means is spaced apart from the user terminal 100 for the next operation. At this time, the proximity state occurs again. In this case, the information providing unit 1810 does not provide the proximity information because the touch means does not intentionally generate the proximity state as the first proximity state after touching the display 1802. When the second proximity state occurs, that is, when the user intentionally generates the proximity state, the information provider 1810 may provide the proximity information.

The mode changing unit 1812 changes the touch mode according to the user's input, and the touch mode may include a proximity touch mode 1820 and a direct touch mode 1822. . Here, the user input may be made through a switch. The switch can be configured as a bell / vibration changeover switch.

In the proximity touch mode 1820, the position image of the touch device in proximity may be displayed on the user terminal 100 or the receiving terminal 102. In addition, the position image of the touch means for touching the user terminal 100 below a predetermined pressure level or area may be displayed on the user terminal 100 or the receiving terminal 102.

In the proximity touch mode, the user terminal 100 recognizes the position of the touch means close to the user terminal 100, and the user terminal 100 touches the position of the touch means according to the touch pressure level and area in the direct touch mode. Recognize the touch as a touch input.

According to an embodiment, the mode changing unit 1812 may change the touch mode when the user terminal 100 and the receiving terminal 102 start interlocking, or may change a user's command, for example, a user's touch, voice data, or image data. Accordingly, the touch mode may be changed. For example, the user may change the touch mode by selecting a menu displayed on the user terminal 100, or may change the touch mode through an image such as a voice or a motion of the user.

The above touch mode change function may be provided only when the interworking of the user terminal 100 and the receiving terminal 102 starts or may be provided to the user terminal 100 regardless of the interworking. In addition, the touch mode change function may be automatically provided when the interlocking is started, or may be provided when the user selects the function after the interlocking. It may be useful to detect when a touch means is near on a small screen such as a smartphone and display the corresponding image on the smartphone. In such a device, it is advantageous to provide a touch mode change function regardless of interworking.

According to another embodiment of the present invention, the user terminal 100 may additionally be provided with an interlocking mode in addition to the proximity touch mode and the direct touch mode. For example, when the user wants to interwork with the receiving terminal 102 while holding the user terminal 100 such as a smartphone or a tablet PC, the user may select an interlocking mode. When the user selects the interlocking mode, the user terminal 100 may start interlocking with the searched receiving terminal 102 by searching for a display device around the user, and may select a proximity touch mode and a direct touch mode when the interworking starts. It may be displayed on the terminal 100.

Although not described above, the selection of the change of the touch mode may be executed by pressing a button provided on the side, the front of the smartphone, etc. in addition to the method of selecting the menu displayed on the user terminal 100.

The controller 1800 generally controls the operations of the components of the user terminal 100.

19 is a block diagram showing the structure of a receiving terminal according to an embodiment of the present invention.

Referring to FIG. 19, the receiving terminal 102 of the present embodiment includes a control unit 1900, an interlocking unit 1902, a transceiver 1904, a signal unit 1906, an image unit 1908, a display unit 1910, It may include an operation execution unit 1912 and a storage unit 1914.

The linkage unit 1902 manages a linkage function with the user terminal 100.

The transceiver 1904 serves as a communication path of the user terminal 100.

The signal unit 1906 may include an information transmitter 1920 and an information receiver 1922.

The information transmitter 1920 transmits image data corresponding to the first image to the user terminal 100.

The information receiver 1922 may receive the position information of the touch means or the selection information of the event execution object transmitted from the user terminal 100.

The image unit 1908 displays the first image through the display unit 1910 and displays a second image corresponding to the received position information of the touch unit together with the first image. According to another embodiment of the present invention, the image unit 1908 may synthesize the first image and the second image, and display the synthesized image through the display unit 1910 as a result of the synthesis.

The display unit 1910 is not limited as long as it displays an image, and may be implemented by, for example, an LCD, an OLED, or a PDP. The display unit 1910 does not have to have a touch function.

The action executor 1912 may execute an action corresponding to the selection information of the event execution object.

The storage unit 1914 stores various data such as a first image, a second image, a composite image, an application program, and the like.

The controller 1900 generally controls the operations of the components of the receiving terminal 102.

On the other hand, the components of the above-described embodiment can be easily identified from a process point of view. That is, each component can be identified as a respective process. In addition, the process of the above-described embodiment can be easily understood in terms of the components of the apparatus.

In addition, the technical contents described above may be embodied in the form of program instructions that may be executed by various computer means and may be recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The embodiments of the present invention described above are disclosed for purposes of illustration, and those skilled in the art having ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention. Should be considered to be within the scope of the following claims.

100: user terminal 102: receiving terminal
110: first image 112: second image
114: the third image

Claims (23)

A decoding unit for decoding the image data received from the receiving terminal;
A touch display unit to display the decoded image;
An information transmitter for transmitting selection information on an event execution object included in the decoded image to the receiving terminal;
An image generator configured to generate a position image of the touch means positioned within a predetermined distance from the touch display unit; And
Including a mode changer for changing the touch mode according to the user input,
The touch display unit displays the location image,
The touch mode,
A first touch mode of displaying the position image; And
A second touch mode that does not display the location image
User terminal. The method of claim 1,
The selection information is
Location information corresponding to a touch input for selecting the event execution object;
User terminal. The method of claim 1,
The selection information is
Control information for executing the event execution object selected according to the touch input
User terminal. The method of claim 1, wherein the image data
Image data for all or part of an image displayed on the receiving terminal.
User terminal. delete The method of claim 1, wherein the image generating unit
To generate a position image of the touch means for touching the touch display unit below a predetermined pressure level or area level
User terminal. delete The shape of the position image representing the position of the touch means is different according to whether the touch means is located at the event execution object or a predetermined position.
User terminal. delete delete Sharing an image with a receiving terminal;
Receiving selection information on an event execution object included in the image;
Transmitting the selection information to the receiving terminal;
Generating a position image of the touch means located within a predetermined distance from the user terminal;
Displaying the generated position image; And
Changing the touch mode according to a user input,
The touch mode,
A first touch mode of displaying the position image; And
A second touch mode that does not display the location image
Terminal interworking method comprising a. The method of claim 11, wherein the selection information is
Location information corresponding to a touch input for selecting the event execution object;
Terminal interworking method. The method of claim 11, wherein the selection information is
Control information for executing the event execution object selected according to the touch input
Terminal interworking method. The method of claim 11, wherein the image is
All or part of the image displayed on the receiving terminal
Terminal interworking method. delete The method of claim 11, wherein the location image is
An image of the touch means for touching the user terminal to a predetermined pressure level or area level or less
Terminal interworking method. delete The method of claim 11,
Changing the shape of the position image representing the position of the touch means according to whether the touch means is located at the event execution object or a predetermined position;
Terminal interworking method further comprising. delete delete delete delete delete

Images