KR20150026773A - User terminal apparatus, method for controlling user terminal apparatus thereof, and expanded display system - Google Patents

Abstract

Provided are a user terminal apparatus which reconfigures a screen according to display characteristics and application characteristics and displays the screen on a user terminal apparatus and an external display apparatus, a control method thereof, and an expanded display system. The control method of a user terminal apparatus includes obtaining information on a display apparatus when the user terminal apparatus is paired with an external display apparatus while an execution screen of an application is being displayed; displaying a first screen of the application based on at least one piece of information on the application that the user terminal apparatus executes and the obtained information on the display apparatus; and transmitting data on a second screen of the application to the display apparatus.

Description

TECHNICAL FIELD [0001] The present invention relates to a user terminal device, a user terminal device control method, and an extended display system,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user terminal, a control method thereof, and an extended display system, and more particularly, to a user terminal, a control method thereof, and an extended display system.

BACKGROUND ART [0002] With the development of related technologies for user terminal devices, contents displayed on user terminal devices have diversified and complicated recently. Such contents can be provided through various applications. There are various kinds of applications ranging from large display screens such as games, moving pictures, cameras, E-books, and applications requiring high power consumption to applications providing a message that is frequently used in existing user terminal devices do.

The existing user terminal device has a limitation on the size of the display screen and the function of displaying the optimal display form convenient for the user to view the screen is lacking according to the display characteristics. In order to solve this problem, there is a technique of reconstructing a screen by dividing the screen into two or more screens to provide a convenient display screen configuration. However, there has been a problem that the content elements can be used only in a limited manner when the mutual inclusion relation and the belonging information are already constituted.

Also, power consumption can not be efficiently managed depending on the type of display or contents.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to provide a user terminal device which reconstructs a screen according to display attributes and application attributes and displays the same on a user terminal device and an external display device, .

According to an aspect of the present invention, there is provided a user terminal comprising: a display unit for displaying an execution screen of an application; A communication unit for performing communication with the display device; Wherein when the display unit is paired with the display device through the communication unit while displaying an execution screen of the application, information about the display device is acquired, information about an application executed by the user terminal device, And a control unit for controlling the display unit to display a first screen for the application based on at least one of information on the first screen and information on the second screen and controlling the communication unit to transmit data of the second screen for the application to the display device do.

The information on the display device may include at least one of a display type, a resolution, a response speed, a color, a gradation, a viewing angle, a color area, a connection angle, and a connection state of the display device.

In addition, the information on the application may include at least one of a category of the application, a color attribute of the application, and a screen update rate of each application.

The control unit may divide the execution screen of the application displayed on the display unit into a plurality of elements based on information on the application, configure the first screen as a part of the divided elements, And the second screen may be configured with the remaining part of the plurality of divided elements.

The control unit may divide an execution screen of an application displayed on the display unit into a plurality of elements based on a partial screen update rate, One screen may be configured, and the second screen may be composed of the remaining elements among the plurality of divided elements.

The control unit divides the execution screen of the application displayed on the display unit into a plurality of elements based on the color attribute of each part, and selects the first color element among the plurality of divided elements, And the second screen may be composed of the elements of the second color among the plurality of divided elements.

The control unit controls the display unit to continuously display an execution screen of the application as the first screen and displays data on a second screen including additional information on the execution screen of the application on the display device So that the communication unit can be controlled.

When the execution screen of the application includes text, the control unit controls the communication unit to transmit data for a second screen including text or an image summarizing the contents of the text to the display device Can be controlled.

The control unit may control the communication unit to transmit data on a second screen including information on the included moving picture content or a caption if the execution screen of the application includes moving picture content, can do.

The control unit controls the display unit to not display the first screen when there is a user input instructing a copy mode, and when the user inputs the copy mode, And to control the communication unit to transmit data on the execution screen to the display device.

The control unit may control the communication unit to transmit data corresponding to a user input to the display device when the display device does not support user input.

According to another aspect of the present invention, there is provided a method of controlling a user terminal, the method comprising the steps of: when the user terminal is paired with an external display device while displaying an execution screen of the application, Acquiring information on the information; And displaying the first screen for the application on the basis of at least one of the information about the application executed by the user terminal and the obtained information about the display device, To the device.

The information on the display device may include at least one of a display type, a resolution, a response speed, a color, a gradation, a viewing angle, a color area, a connection angle, and a connection state of the display device.

In addition, the information on the application may include at least one of a category of the application, a color attribute of the application, and a screen update rate of each application.

Dividing an execution screen of the displayed application into a plurality of elements based on information about the application; And configuring the first screen with a part of the plurality of divided elements and configuring the second screen with the remaining part of the plurality of divided elements.

The dividing step may divide the execution screen of the displayed application into a plurality of elements based on the partial screen update rate, and the constructing step may include a step of dividing the moving picture among the plurality of divided elements The first screen may be composed of the elements including the divided elements, and the second screen may be composed of the remaining elements of the divided elements.

The dividing step may divide an execution screen of the displayed application into a plurality of elements based on color attribute of each part, and the constituting step may include dividing the execution screen of the displayed application into a plurality of elements, The first screen may be composed of the elements of the first color and the second screen may be composed of the elements of the second color among the plurality of elements.

The transmitting step may continuously display the execution screen of the application as the first screen and transmit data on the second screen including the additional information on the execution screen of the application to the display device .

In addition, when the execution screen of the application includes text, data for a second screen including a text or an image summarizing the contents of the included text may be transmitted to the display device.

If the execution screen of the application includes moving picture contents, the display apparatus can transmit data on the second screen including the information on the included moving picture contents or the caption.

The transferring step may include a step of, when there is a user input instructing a copy mode, displaying the first mode on the execution screen of the application when a user input indicating the copying mode has been performed To the display device.

The method may further include transmitting data corresponding to a user input to the display device when the display device does not support user input.

According to another aspect of the present invention, there is provided an extended display system including: a display device that is paired with a user terminal and displays data of a second screen received from the user terminal; And acquiring information on the display device when it is paged with the display device while displaying an execution screen of the application, and based on at least one of information on the executing application and information on the obtained display device And a user terminal device for displaying a first screen for the application and transmitting the data of the second screen for the application to the display device.

According to various embodiments of the present invention as described above, it is possible to implement various user scenarios using a wide screen by interlocking the user terminal device and the display device, and it is possible to provide an optimal And the content can be displayed at a low power according to the power consumption characteristics of the display and the application.

Figure 1 illustrates an extended display system, in accordance with an embodiment of the present invention;
2 and 3 are block diagrams illustrating a configuration of a user terminal according to an embodiment of the present invention,
4 is a block diagram for explaining a module stored in a storage unit of a user terminal device,
5 is a diagram for explaining an attribute of an application according to a category,
6A and 6B are views for explaining a screen configuration according to a partial screen update rate according to an embodiment of the present invention;
FIGS. 7A through 7C are views for explaining a screen configuration according to a color attribute according to an embodiment of the present invention; FIG.
8A to 8C are views for explaining various forms of separating and reconstructing a screen according to an embodiment of the present invention;
FIGS. 9A and 9B are views for explaining a case of separating screens of a moving image playback application according to an embodiment of the present invention;
FIG. 10 is a view for explaining a case of separating a screen of an E-book according to an embodiment of the present invention; FIG.
FIG. 11 is a diagram for explaining a case where a web browser screen is displayed in a summary manner according to an embodiment of the present invention; FIG.
12 is a diagram for explaining a case where the contents of an E-book are summarized and displayed according to an embodiment of the present invention;
FIG. 13 is a diagram illustrating a graphical display of contents of a schedule application according to an embodiment of the present invention; FIG.
FIG. 14 is a view for explaining a case of displaying photograph-related meta information according to an embodiment of the present invention;
FIG. 15 is a diagram for explaining a case of displaying information on a moving picture or subtitles according to an embodiment of the present invention; FIG.
16 is a diagram for explaining a case of displaying help related to a web browser screen according to an embodiment of the present invention;
17 is a view for explaining a case of displaying help related to contents of an E-book according to an embodiment of the present invention,
18 is a diagram for explaining a case of displaying composition information and the like upon photographing according to an embodiment of the present invention,
19A and 19B are diagrams for explaining a copy mode in a web browser according to an embodiment of the present invention;
FIG. 20 is a diagram for explaining a copy mode in an E-book according to an embodiment of the present invention; FIG.
21 is a diagram for explaining touch input support according to an embodiment of the present invention;
FIG. 22 is a view for explaining screen editing function support according to an embodiment of the present invention; FIG.
23 is a view for explaining a case where a surrounding dialogue is recognized and photographed in a real-time STT (sound-to-text) form according to an embodiment of the present invention;
24A and 24B are views for explaining a case of displaying a widget according to an embodiment of the present invention;
25 is a diagram for explaining a case of displaying a message or an email widget according to an embodiment of the present invention;
26 and 27 are flowcharts for explaining a control method of a user terminal according to various embodiments of the present invention;
28 is a sequence diagram for explaining the operation of the extended display system according to an embodiment of the present invention;
29A and 29B are views and block diagrams for explaining a user terminal apparatus having a plurality of display units according to an embodiment of the present invention,
30A to 31C are views for explaining a case where a display device is configured in a case of a user terminal device according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating an extended display system, in accordance with an embodiment of the present invention. Referring to FIG. 1, the extended display system includes a user terminal device 100 and a display device 200.

The display device 200 receives data from the user terminal device 100 and displays a screen corresponding to the received data. The display device may be a smart watch, a smart phone, or a smart glass as shown in FIG. 1, but it may be a variety of devices having a display unit such as an electronic paper (E-paper), an electronic photo frame, But it is not limited thereto. In addition, the display device 200 such as a smart phone including the control unit may perform the functions performed by the control unit of the user terminal device 100 described below separately.

When the execution screen of the application is paired with the external display device 200 while the execution screen is being displayed, the user terminal device 100 acquires information about the display device 200. Based on at least one of the information about the executing application and the information about the acquired display device 200, the user terminal device 100 displays the first screen for the application. Also, the user terminal device 100 transmits the data of the second screen for the application to the display device 200.

By using the user terminal device 100 and the display device 200 in a pair by using the extended display system as described above, the user can obtain effects such as enlargement of the display screen, maximization of visibility, and efficiency of power consumption will be.

2 is a block diagram schematically illustrating the configuration of a user terminal device 100 according to an embodiment of the present invention. Referring to FIG. 2, the user terminal 100 includes a display unit 110, a communication unit 120, and a controller 170. Hereinafter, it is assumed that the user terminal device 100 is a smart phone. However, the user terminal device 100 may be implemented as a device capable of executing an application such as a tablet PC, but is not limited thereto.

The display unit 110 displays a screen to be executed in various user terminal devices 100 including an application execution screen. The display unit 110 may be designed with various display panels. That is, the display unit 110 may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an electronic paper (E-paper), a plasma display panel (PDP), a Vacuum Fluorescent Display (VFD), a Field Emission Display (FED), and an Electro Luminescence Display (ELD). The display panel is mainly made of a light emitting type, but does not exclude a reflective type display. Also, the display unit 110 may be implemented as a flexible display, a transparent display, or the like.

The communication unit 120 transmits and receives data to and from the external display device 200. The communication unit 120 is composed of a wireless or wired communication module and performs a wireless or wired communication function with the display device 200. Specifically, the communication unit 120 receives the information of the display device 200 and transmits data on the second screen to be displayed on the display device 200 to the display device 200.

The control unit 170 controls the overall configuration of the user terminal device 100. Particularly, when the display unit 110 is paired with the display device 200 through the communication unit 120 while the display unit 110 displays an execution screen of the application, the control unit 170 acquires information about the display device 200, The display unit 110 controls the display unit 110 to display a first screen for the application on the basis of at least one of the information about the application executed by the terminal apparatus 100 and the information about the obtained display apparatus 200, And control the communication unit 120 to transmit the data of the second screen to the display device 200. [

Specifically, the control unit 170 divides the execution screen of the application displayed on the display unit 110 into a plurality of elements based on the information about the application, forms a first screen as a part of the plurality of divided elements, And the second screen may be composed of the remaining part of the plurality of divided elements. In one embodiment, the control unit 170 divides the execution screen of the application displayed on the display unit 110 into a plurality of elements based on the partial screen update rate, and, among the plurality of divided elements, The first screen may be composed of the elements, and the second screen may be composed of the remaining elements of the plurality of elements. In the case of a moving picture, since a screen update rate must be fast enough, it is preferable that a moving picture is displayed on a display panel such as an OLED. On a display panel, such as an electronic paper (e-paper) It is preferable that a screen with a slow screen update rate such as text excluding the display is displayed. In another embodiment, the control unit 170 divides the execution screen of the application displayed on the display unit 110 into a plurality of elements based on the partial color attribute, and selects an element of the first color among the plurality of divided elements And a second screen may be composed of the elements of the second color among the plurality of divided elements. There is a display type in which power consumption is increased when displaying a screen with many bright displays close to white depending on the properties of the display. For example, since OLEDs have self-luminescence characteristics, power consumption is large when there are many bright displays close to white. Accordingly, the controller 170 forms the second screen with the second color as white based on the color attribute of each part, and the second screen is composed of the elements with a lot of bright display. In the OLED, One screen can be configured and displayed.

On the other hand, the control unit 170 controls the display unit 110 to continuously display the execution screen of the application as the first screen, and displays the data on the second screen including the additional information on the execution screen of the application on the display device 200 to the communication unit 120. For example, when a user searches for additional information on an execution screen of an application using only the user terminal device 100, it may be inconvenient to stop execution of the application. If this additional information is displayed on the external display device 200, the user will be able to continue executing the application on the user terminal device 100 without interruption. In one embodiment, when the execution screen of the application includes text, the control unit 170 transmits data for a second screen including text or an image summarizing the contents of the included text to the display device 200 The communication unit 120 can be controlled. In another embodiment, when the execution screen of the application includes moving picture contents, the controller 170 transmits information about the included moving picture contents or data on the second screen including the subtitles to the display device 200 The communication unit 120 can be controlled.

The control unit 170 controls the display unit 110 to not display the first screen when there is a user input for instructing the copy mode and displays the application when the user inputs the copy mode The communication unit 120 can control the display unit 200 to transmit data on the execution screen of the display unit 200. [ For example, when an electronic paper (E-paper) using reflected light is used as the display device 200, the electronic paper (E-paper) can display characters and pictures without consuming electricity after the screen is changed The user can move the screen displayed on the display unit 110 to an electronic paper (e-paper) having low power consumption by using a copy mode.

The control unit 170 may control the communication unit 120 to transmit data corresponding to the user input to the display device 200 when the display device 200 does not support user input. For example, in order to switch the screen of the display device 200 that does not support scroll input, after receiving the scroll input from the user terminal device 100, the controller 170 displays the scroll input on a display device The data corresponding to the scroll input applicable to the display device 200 can be transferred to the display device 200.

The user terminal device 100 as described above can display a screen in an optimal form so that the user does not inconvenience a view of the screen using the display device 200 and various additional information can be displayed on the display device 200. [ And the content can be displayed with low power according to the power consumption characteristics of the user terminal 100 and the display device 200. [

3 is a block diagram illustrating a configuration of a user terminal 100 according to an exemplary embodiment of the present invention. 3, the user terminal 100 includes a display unit 110, a communication unit 120, a storage unit 130, an interface unit 140, an audio output unit 150, a camera unit 160, (Not shown).

The display unit 110 displays a screen to be executed in various user terminal devices 100 including an application execution screen. When the user terminal device 100 is used alone, the application execution screen is displayed regardless of the display type of the display unit 110 or the like. Therefore, depending on the type of the displayed screen, the display unit 110 having a specific display type may not be appropriate in terms of power consumption and the like. Therefore, it is desirable to configure the first screen to be displayed on the display unit 110 in consideration of the characteristics of the display unit 110 when the screen is reconstructed. The characteristics of the display unit 110 such as the display type will be described later.

The communication unit 120 transmits / receives information to / from the display device 200. The communication unit 120 is composed of a wireless or wired communication module and performs a wireless or wired communication function with the display device 200. Specifically, the communication unit 120 receives the information of the display device 200 and transmits data on the second screen to be displayed on the display device 200 to the display device 200.

The communication unit 120 may use a High Definition Multimedia Interface (HDMI), a Low Voltage Differential Signaling (LVDS), a Local Area Network (LAN), or the like as a wired communication method. When the communication unit 120 connects with the display device 200 through HDMI which is one of the specifications of the non-compression type digital video / audio interface, a standard method of exchanging attributes between a source and a sink supported by the HDMI itself There are advantages to be able to use. For example, through a defined method between the InfoFrame based on CEA861 and the Source and Sink devices using the E-EDID, the Sink device can provide the resolution and other display related attributes of the Sink device to the Source device. Similarly, even when the user terminal device 100 and the display device 200 are faced with each other via the HDMI, information related to the display device 200 can be recorded in the E-EDID and can be provided through the DDC channel, The display device 200 can receive the physical connection status of the display device 200 and the user terminal device 100 and the display attribute information of the display device 200. [ LVDS is a signaling standard for transmitting image data from a video device to a computer monitor. It is used in Flat Panel Display Link (FPD) links in flat panel or open LVDS display interface (OpenLDI) standard. When communicating through the wired connection of the LVDS system, the communication unit 120 can check the physical connection state of the wired LVDS of the display device 200, and the communication unit 120 may transmit the reserved or undefined It can be used to exchange display attribute information by newly defining a data portion. Specifically, the LVDS interface is divided into JEIDA and VESA, which may vary according to the number of bits and the number of pairs, and the LVDS interface can be used to exchange display-related attributes using reserved data portions. When the user terminal device 100 and the display device 200 are paired with each other via the wired LAN, information on whether or not the communication connection is established with a predetermined protocol for mutual recognition between the user terminal device 100 and the display device 200 Should be able to be transmitted and received. Similarly to the method of defining the protocol, a purpose of exchanging data through the communication unit 120 such as display attribute information can also be defined. For this purpose, a corresponding protocol can be defined through an application layer or a presentation layer at a higher level in the OSI 7-Layer Model (Open System Interconnection 7 Layer Model).

The communication unit 120 can use various methods such as NFC (Near Field Communication), wireless LAN (Wireless LAN), IR (InfraRed) communication, Zigbee communication, WiFi, and Bluetooth. NFC is a technology for wireless communication in a very short distance and is used in various services such as traffic, tickets, and payment. Currently supported NFC data communication speed is 424 kbit / s. The NFC reader and the NFC tag may be embedded in the user terminal device 100 and the display device 200 for pairing and exchange of property information between the user terminal device 100 and the display device 200, It is possible to check whether the user terminal 100 and the display device 200 are in contact with each other through the natural operation in which the NFC reader and the NFC tag are in contact with each other, The NFC reader built in the user terminal device 100 reads the information and the display attribute information of the display device 200 can be received by the communication unit 120. [ When the user terminal device 100 and the display device 200 are paired with each other via the wireless LAN, information on whether or not communication is established with a predetermined protocol for mutual recognition between the user terminal device 100 and the display device 200 Should be able to be transmitted and received. Similar to the wired LAN method described above, a protocol can be defined using an application layer or a presentation layer at a higher level in the OSI 7 layer model, and the communication unit 120 can transmit and receive data using the defined protocol.

Although various wireless and wired communication methods that the communication unit 120 can use as described above have been described, the present invention is not limited to the communication methods described above.

The storage unit 130 stores various programs, modules, and data necessary for the operation of the user terminal device 100. FIG. 4 is a block diagram showing a part of the module stored in the storage unit 130. FIG. As shown in FIG. 4, the storage unit 130 may include a display attribute management module 131, an application attribute management module 132, and a screen reconstruction module 133.

The display attribute management module 131 recognizes the display type, resolution, response speed, color, gradation, viewing angle, color area, connection angle, connection status, etc. of the display device 200 paired with the user terminal device 100 . The display attribute management module 131 includes a connection display type recognition module 131-1, a connection display attribute recognition module 131-2, an inter-display connection angle determination module 131-3 and a display connection state determination module 131- 4) may be included.

Specifically, the connected display type recognition module 131-1 is used to recognize the display type of the display device 200 that is paired based on the operation mode. Since the power consumption characteristics are different from each other depending on the display type, it is necessary to distinguish the display method according to the display type. For example, in the case of a liquid crystal display (LCD), one pixel is made of a liquid crystal in which two transparent electrodes are connected, and there is a polarization filter perpendicular to each other on both sides. Normally, a certain arrangement of the liquid crystal passes light but when the voltage is applied, the arrangement of the liquid crystal is fixed and the light can not pass and is blocked. Therefore, the LCD exhibits a constant power consumption characteristic irrespective of the screen to be displayed. As another example, an organic light emitting display (OLED) is a thin film light emitting diode in which the light emitting layer is an organic compound. Emitting characteristics, OLEDs are characterized in that they consume a large amount of electric power in the case of many bright displays close to white. As another example, electronic paper (E-paper) is a display technology in which the characteristics of ink common to paper are applied. Unlike traditional flat-panel displays that use a backlight to brighten pixels, electronic paper (E-paper) uses reflected light like ordinary paper. Due to the principle of operation having a bi-stable characteristic, after the screen is changed, the text and the picture can be displayed without electricity consumption. In addition, electronic paper (E-paper) has a feature that it can be folded or rolled differently than a flat panel display. Electronic paper (E-paper) corresponds to the display that can be used the most among existing displays. Electronic paper (E-paper) has a wider viewing angle than an LCD, so it can read letters easily from a weak angle. However, due to the nature of reflection, it is impossible to use backlight in electronic paper (E-paper), and there is a disadvantage that the reaction rate is slow. As another example, in the case of a plasma display (PDP), since there is no afterimage due to the use of electric discharge of a plasma, a smooth screen is provided, so that fatigue of eyes is small and the price is low in comparison with the number of inches. However, the PDP has a disadvantage of relatively high power consumption, heat generation, and thickness.

The connected display attribute recognition module 131-2 is a module used for analyzing common property information of displays such as resolution, response speed, color, and gray level of the display device 200 to be paired. The screen reconstruction and display method must be changed according to the display attributes of the display device 200, and thus it is necessary to use the connection display attribute recognition module 131-2. For example, in the case of response time, the OLED has a response speed in microseconds, and the LCD has a response time of 2 to 30 ms in response to the improvement of the reaction speed. ) Has a reaction rate of about 250 ms. Therefore, in the case of an image requiring a screen update of about 30 fps, such as a general moving image, it is difficult to display on an electronic paper (E-paper). Even in color and gradation, electronic paper (E-paper) has mostly grayscale and supports only some colors.

The inter-display connection angle determination module 131-3 is a module that can be widely used when the user terminal device 100 and the display device 200 are particularly wired. In the case of the wired connection, a method may be used in which a portion connected to the piezoelectric element is constituted by a piezoelectric element, and the stress due to the bent angle is measured as a resistance value to grasp the connection angle. However, even in the case of wireless connection, since the angle of connection can be determined using a sensor or the like, it is not a module used only in a wired connection. It is necessary to grasp the connection angle because the scenario that can be displayed using the user terminal device 100 and the display device 200 varies depending on the connection angle. For example, when the connection angle is 0 °, it can be assumed that the display portion 110 and the display portion of the display device 200 are in contact with each other. If the size of the display is different, simple information or the like may be displayed using the remaining portion without overlapping. As another example, there may be an embodiment in which the copy mode is executed when the connection angle is around 60 degrees. As another example, there may be cases where the connection angle is 180 degrees. In a typical embodiment, the user terminal 100 and the display device 200 will be paired with a connection angle of mostly 180 degrees. Exemplary embodiments described below are described on the assumption that the connection angle is 180 [deg.], But the connection angle is not necessarily 180 [deg.]. As another example, there may be a scenario in which the display unit 110 and the display unit of the display device 200 are placed on a table in a state in which they are opposite to each other when the connection angle is about 270 degrees. There may be a case in which one user sees only one screen on a laundering ceremony, but there may be cases where two or more users sit face to face with each other. As another example, when the connection angle is 360 °, the display portion 110 and the display portion of the display device 200 may both be exposed as opposed to the case where the connection angle is 0 °.

The display connection state determination module 131-4 is a module that can be used to determine whether the user terminal device 100 and the display device 200 are paired or not. The determination method may vary depending on what type of communication method is used in the communication unit 120. [

The application attribute management module 132 is a module used for recognizing and determining the category of the application executed in the user terminal device 100, the color attribute for each part, and the screen update rate for each part. The application property management module 132 may include an application type identification module 132-1, a partial screen update rate analysis module 132-2, and a partial color property analysis module 132-3.

Specifically, since the required display characteristics are different depending on the category of the application to be executed, the application type identification module 132-1 is used to identify the category of the application executed in the user terminal device 100. [ 5 is a diagram for explaining attributes of an application according to a category. The nine categories of various application categories are Home & Lock Screen, Web Browser, E-book, Message, Camera, Video Player, Calendar, Gallery and Game. As shown in FIG. 5, battery consumption is different for each category, and possible screen reconstruction scenarios are different. For example, referring to FIG. 5, it can be seen that a game application consumes about five times as much battery power per minute as a home & lock screen application.

The partial screen update rate analysis module 132-2 is a module used to reconstruct the screen by separating the application execution screen according to the screen update cycle. The reason why the screen update rate analysis is necessary is that there is a device that can not display contents requiring a fast screen update rate depending on display characteristics. For example, if the response time is as slow as 250 ms, such as electronic paper, contents requiring fast update rate such as moving images can not be displayed. 6A shows that an application execution screen including a portion 610 in which a moving image requesting a fast update rate is displayed and a portion 620 in which text not to be updated is displayed is displayed in the user terminal device 100 FIG. The application execution screen can be divided into two parts 610 and 620 using the partial screen update speed analysis module 132-2. As shown in FIG. 6B, a portion 610 displaying a moving image requiring a fast update rate is displayed on the user terminal 100 using an OLED or the like having a fast response speed, and is not required to be updated The portion of the text to be displayed 620 is suitably adapted to transmit and display data on the display device 200, such as electronic paper (E-paper), once it has been updated, without additional power consumption.

The partial color property analysis module 132-3 can be used to provide a screen reconstruction criterion by obtaining an area-by-area color distribution histogram by color composition analysis and providing a power consumption criterion according to the color distribution. The reason why the analysis of the color attribute of each part of the screen to be displayed is necessary is that the power consumption differs according to the display color constituting the screen depending on the display characteristic. Particularly, when a self-emission type display such as an OLED is applied, the color property of each part should be considered. FIG. 7A shows an example of color histogram analysis for each region used for analyzing the color attribute of each region. As shown in FIG. 7A, color analysis can be performed on a monochrome screen, but analysis can be performed even when various colors are included. FIG. 7B shows screens 710 and 720 on which an application is executed in the user terminal device 100 before the screen is reconfigured. The portion 710 where the moving image is displayed is determined to be an area having a large distribution of dark colors, the portion 720 where the text is displayed has black text, and the background of the text is white, . The screens 710 and 720 can be divided and displayed on the user terminal device 100 and the display device 200 as shown in FIG. 7C with the result of analyzing using the partial color property analysis module 132-3. For example, the display unit 110 of the user terminal device 100 displays a portion 710 in which a moving image, which is a black color screen having a small power consumption, is displayed when the OLED is configured, In the portion 720 where the text is displayed, it is desirable to transmit data to the display device 200 using an electronic paper (E-paper) or an LCD which does not increase power consumption of the white series and display the data, Do.

The screen reconstruction module 133 is used to reconstruct a screen to be displayed in the user terminal device 100 and the display device 200 using a method of separating elements of an application being executed in the user terminal device 100 . The screen reconstruction module 133 includes an application element separation module 133-1, an application page division module 133-2, an operation layout pattern application module 133-3 and an adaptive text layout module 133-4 .

Specifically, the application element separation module 133-1 is a module that separates one application into each element. In the case of an application displaying a structured document such as a web page, there is no need to divide a separate element. However, in the case of a general application, it is necessary to separate parts according to attributes of the screen. The separation criterion may be provided through the application property management module 132 described above. The method of separation may include a method of detecting through a segmentation process, a method of parsing the layout of an app, and the like.

The application page divider module 133-2 may be configured to divide the display device 200 into two categories according to the characteristics of the display device 200 paged to the user terminal device 100, And if the display device 200 is suitable for the page interface, performs page division. For example, in a case where a screen for displaying a text is provided using the scroll in the user terminal device 100, the screen for displaying the text is transmitted to the display device 200 using the electronic paper (E-paper) Can be displayed. Due to the characteristics such as the reaction rate, it is difficult to apply the scroll method to the electronic paper (E-paper). In this case, the application page division module 133-2 may divide the text on a page-by-page basis to provide a display device 200 using electronic paper (E-paper) so that a scroll interface does not occur.

The dynamic layout pattern application module 133-3 may divide each screen into a plurality of sublayers by using the application element separating module 133-1 in an adaptive manner Which is a module that provides control functions that can be performed by The dynamic layout pattern application module 133-3 may dynamically display the screen separately, and may also include a function of fitting the screen layout according to the screen size. For example, referring to FIG. 8A, an execution screen displayed on the user terminal 100 may be divided into five elements 810, 820, 830, 840 and 850. 8B shows a case where the execution screen is dynamically screen-separated. In the user terminal 100, three elements 810, 830 and 850 of the separated elements are displayed and the remaining two elements 820 and 840 can be displayed in the display device 200. [ 8C shows a case where the execution screen is dynamically separated and fitting is performed in accordance with the screen size. In the user terminal device 100, the three elements 820 ', 830', and 840 'are displayed by changing the screen size according to the arrangement structure before reconfiguration and the size of the display unit 110. [ In the display device 200, the two elements 810 'and 850' are enlarged from the original size in accordance with the size of the screen. In the above description, only the size of the element is changed according to the size of the screen. However, it is also possible to change the size of the element based on other characteristics such as the display resolution, but the present invention is not limited thereto.

The adaptive text layout module 133-4 may be used to display a web page including text in the display device 200. If an enlargement / reduction instruction is given or a screen size is changed, a text flow Which is a module that provides automatic control of The scenario in which the adaptive text layout module 133-4 is used together with the application page division module 133-2 described above may be assumed. For example, when a text screen displayed on the user terminal device 100 is transmitted to the display device 200 for display, the font size may be changed to a font size suitable for the display device 200, The adaptive text layout module 133-4 can provide a function of converting the text so as not to overflow the text.

The interface unit 140 allows the user terminal device 100 to exchange commands with a user. For example, this may be a touch screen, a keypad, or the like. According to an embodiment of the present invention, when the display device 200 does not support user input, the interface unit 140 receives user input from the user terminal device 100 and transmits corresponding data to the communication unit 120. [ To the display device 200 through the display device 200. [ In addition, when it is inconvenient to directly edit the screen by touching the screen of the display device 200 because the screen of the display device 200 is small, the screen of the display device 200 may be edited by receiving a screen editing command of the user have.

The audio output unit 150 may output a voice signal of an application executed in the user terminal device 100. [

The camera unit 160 allows an application providing a photographing function and a moving picture photographing function to photograph. In addition, the camera unit 160 may perform motion recognition and the like of the user together with the interface unit 140. For example, when the user performs a hand operation corresponding to the copy mode, the camera unit 160 may photograph the hand operation of the user and provide it to the interface unit 140 or the control unit 170 have.

The control unit 170 controls the display unit 110, the communication unit 120, the storage unit 130, the interface unit 140, the audio output unit 150, and the camera unit 160. The control unit 170 may include a ROM (Read Only Memory) and a RAM (Random Access Memory) for storing a module and data for controlling the CPU and the user terminal device 100. In this case, the display attribute management module 131, the application attribute management module 132, and the screen reconstruction module 133 described above may be stored in the controller 170.

The control unit 170 acquires information about the display device 200 when the display unit 110 is paired with the display device 200 through the communication unit 120 while displaying the execution screen of the application, The display unit 110 controls the display unit 110 to display a first screen for the application based on at least one of the information about the application executed by the mobile terminal 100 and the information about the obtained display device 200, And control the communication unit 120 to transmit the screen data to the display device 200. [

Specifically, the control unit 170 divides the application execution screen into a plurality of elements based on the information about the application executed in the user terminal device 100 acquired using the application element separation module 133-1 . A first screen is formed as part of the divided elements, and a second screen is composed of remaining elements.

In an embodiment of the present invention, as shown in FIG. 9A, an execution screen composed of a portion 910 including a moving picture and a portion 920 including a text may be displayed on the user terminal device 100. Before dividing the execution screen into a plurality of elements, it is necessary to recognize the information about the display type and the attribute. For example, assume that the user terminal apparatus 100 shown in FIG. 9B is a device in which the display unit 110 is an OLED, and the display device 200 is an apparatus made of an E-paper. OLEDs are characterized by high power consumption in white representation, and electronic paper (E-paper) is characterized by difficulty in displaying contents such as moving images with fast transitions. Accordingly, the controller 170 can divide the application execution screen into the element 910 including the moving picture and the element 920 including the remaining text, based on the color attribute and the screen update rate. The controller 170 configures the first screen with the element 910 including the moving picture and controls the display on the user terminal 100. The second screen is composed of the element 920 including the text, It is possible to control the communication unit 120 to transmit data for the two screens to the display device 200. [

In another embodiment of the present invention, FIG. 10 shows a case where one E-book screen is divided and displayed between the user terminal 100 and the display device 200. The e-book screen may include a picture part 1010 in which a color is included and a part 1020 in which a text is displayed. In addition, the picture part 1010 may be substituted for the picture part 1010, or may include moving picture contents, interactive contents, etc. along with pictures. 10, it is assumed that the display unit 110 is an OLED device and the display device 200 is an electronic paper (E-paper). In such a case, it is appropriate to display on the OLED display for color display or moving picture display, and to display on the electronic paper (E-paper) for which power consumption is low, in a text portion in which many white series are arranged. Accordingly, the screen of the E-book is divided to constitute the first screen with the element 1010 including the picture part, so that the first screen is displayed in the user terminal device 100, and the element 1020 ) So that the second screen is displayed on the display device 200 is an efficient screen reconstruction method.

According to an embodiment of the present invention, the control unit 170 controls the display unit 110 to continuously display an execution screen of an application executed in the user terminal device 100 as a first screen, And to control the communication unit 120 to transmit the data for the second screen including the information to the display device 200. [

11 is a view for explaining an embodiment for displaying a summary of a web browser screen. 11, a web browser application is executed in the user terminal 100, and a web page screen 1110 is displayed on the display unit 110. As shown in FIG. The second screen 1120 to be displayed on the display device 200 by summarizing the contents of 'Geneva', 'Motor Show', etc., which is the main contents of the web page screen 1110, recognizes the contents of the web page screen 1110 Lt; / RTI > Natural text processing techniques can be used to recognize and summarize text. For the user's visibility, for example, there may be a method of displaying the font size largely in proportion to the number of times a word appears. The update period of the information constituting the second screen 1120 may be updated every time the web page is refreshed in the user terminal device 100 or may be updated every predetermined time period. The readability of the user can be improved through the embodiment shown in FIG.

12 is a view for explaining an embodiment for displaying the contents of the E-book in summary. Referring to FIG. 12, an E-book execution screen 1210 is displayed on the user terminal device 100 and a screen in which the content of the E-book execution screen 1210 is summarized into a few lines in the display device 200 (1220) is being displayed. The update period of the summary screen 1220 may be updated at every page change of the E-book executed in the user terminal device 100. If the E-book application executed in the user terminal device 100 is a type that scrolls the content, a method of updating the E-book application every predetermined period will be possible. In order to recognize the contents of the e-book, natural language processing methods and the like may be applied. In FIG. 12, only a screen 1220 in which the contents of the E-book is summarized by sentences is displayed, but it may be expressed by a picture or an icon so that the user can easily recognize the screen 1220.

13 is a diagram for explaining an embodiment of graphically displaying contents of a schedule application. 13, a screen 1310 for executing a calendar application including a schedule is displayed in the user terminal device 100. In the display device 200, the entire schedule related to the content of the schedule is displayed in a second form A screen 1320 is displayed. In order to organize the schedule in cartoon form, information about the nature of the meeting recorded in the calendar application, urgent schedule information, and the like may be considered in addition to the contents of the schedule. The second screen 1320 does not necessarily have to be composed only of a cartoon, and a second screen may be formed using a photograph taken by the camera unit 160 on the date when the second screen 1320 is scheduled.

14 is a diagram for explaining an embodiment for displaying photograph-related meta information. The photograph-related meta information may include the resolution, size, format, shooting date, photographing device information, and the like of the photograph. 14, an execution screen 1410 of a gallery application for displaying a photograph is displayed on the user terminal 100, and the display device 200 displays meta information of a photograph displayed on the first screen 1410 A second screen 1420 is displayed. The update period of the meta information displayed on the second screen 1420 is preferably updated whenever the user terminal 100 selects an individual photograph. In another embodiment, when the user inputs the user information in the display device 200, if a specific meta information value is selected and retrieved from the second screen 1420, The first screen 1410 may be configured and displayed. For example, in the case where the photographing time of the meta information is retrieved on May 15, if there are four photographs taken on May 15 of the photographs stored in the user terminal device 100, The first screen 1410 arranged in a date may be configured and displayed.

According to an embodiment of the present invention, when the execution screen of the application executed in the user terminal device 100 includes the moving picture content, the control unit 170 displays the information about the included moving picture content or the second It is possible to control the communication unit 120 to transmit data on the screen to the display device 200. [ At this time, the information on the moving image content may include plot information, information on the director or the actor, information on the date of the next moving image content update when the content is a series, and the like. For example, in the method of acquiring plot information, the user terminal device 100 may acquire and provide information by accessing the Internet in the background, or may use the information included in the video content itself. FIG. 15 is a diagram for explaining an embodiment for displaying information or subtitles about a moving picture; FIG. Referring to FIG. 15, in the user terminal device 100, an execution screen 1510 of a moving picture player application capable of reproducing moving picture contents is displayed. In the display device 200, the subtitles of the moving picture contents displayed on the user terminal device 100 are displayed on the second screen 1520. Since the electronic paper (E-paper) having the slowest response speed has a screen update rate sufficient to display the subtitles, the embodiment for displaying the subtitles can be implemented in the display device 200 of all display types Do.

In another embodiment of the present invention, when an application for displaying a web page in the user terminal device 100 is being executed, contents that can help read the web page can be displayed on the display device 200. [ When it is desired to search for an interpretation of a related term while reading a web page using the user terminal device 100, the user has to stop the web browsing in order to convert the screen on which the web page is displayed to a screen for term search I have to accept it. However, if it is possible to search for related terms in the display device 200, the user can obtain related information while browsing the web without interruption. 16 is a view for explaining an embodiment for displaying help related to a web browser screen. The execution screen 1610 of the web browser application is displayed on the user terminal device 100 and the display device 200 displays a second web page including the glossary information on the 'concept car' A screen 1620 is displayed. The part requiring the commentary may be automatically selected from the contents of the web page 1610 or may be selected by the user through the interface unit 140 through the touch input from the first screen 1610. [ For example, if the user selects the 'Geneva International Motor Show' portion of the contents of the first screen 1610, information on 'Geneva International Motor Show' will be displayed on the second screen 1620. Information necessary for configuring the second screen 1620 may be acquired in a dictionary application stored in the user terminal device 100 or may be acquired through an Internet connection. In the case of automatically selecting a portion requiring a glossary, the information update cycle may be updated every time the web page 1610 is refreshed in the user terminal device 100, or after a predetermined period of time, And may be updated to display commentary information. When the user selects a portion requiring a glossary, the information update period may be set so that the update is not performed until the input of the next user is made.

17 is a diagram for explaining an embodiment for displaying help related to the contents of the E-book. 17, an execution screen 1710 of the E-book application is displayed on the user terminal 100, and English dictionary information on words displayed on the E-book is provided on the display device 200. [ In the second screen 1720, 'Swing' among the words displayed in the E-book and 'Swing' corresponding to Korean are displayed together. The information on the E-book constituting the second screen 1720 may include a dictionary, a related term commentary, a bookmark, an emphasis, a memo, and the like. Parts that require glossary, dictionary search, etc. can be automatically selected from among the contents of the e-book, and the user can select a portion requiring information. In the case of a dictionary search, the database may be customized according to a personalized database using an existing dictionary search history. The period of the information update may be updated each time the page of the E-book is switched in the user terminal device 100. [ However, if it is more convenient for the user to continue to display regardless of page switching such as bookmark, highlight, and memo, an update period may be set so as not to update until a specific user input is made.

Performance of the camera device mounted on the camera unit 160 of the recent user terminal device 100 is gradually improving and it is also common for users to take pictures using the user terminal device 100. [ However, since the general user does not have the expertise in the angle and composition of the photographing, the information about the photographing angle and the composition can be provided, thereby providing a useful user experience to the user. 18 is a diagram for explaining an embodiment for displaying composition information and the like upon photographing. The user terminal device 100 displays a photographing screen 1810 shown through the camera unit 160. [ In the case where the composition information is provided by the user terminal device 100, the photographing screen 1810 and the composition information screen must be alternately displayed. Therefore, there is an inconvenience that the user must depend on the memory capacity of the user. By providing the sketch screen 1820 for the same place on the display device 200, the user terminal device 100 allows the user to more easily take a picture corresponding to the recommended composition. In particular, in the case of a famous tourist spot, it is also possible for the user to be provided with information on a representative photograph composition of a tourist spot currently located using a GPS device that can be included in the user terminal device 100.

In another embodiment, in the case of executing the message application in the user terminal device 100, the second screen of the display device 200 may display the previous conversation history, anniversary or event information with the conversation partner. By displaying the previous conversation history and the like on the display device 200, it is possible to help the user to have a smooth conversation with the other party, and to prevent the user from making a mistake during conversation.

According to an embodiment of the present invention, when there is a user input indicating a copy mode, the controller 170 controls the display unit 110 to not display the first screen, It is possible to control the communication unit 120 to transmit data on the application execution screen at the time of user input to the display device 200. [

19A is a diagram for explaining a case where there is a user input indicating a copy mode (Copy Mode) in the web browser. Referring to FIG. 19A, in order to instruct the copy mode, the user performs an operation of browsing over the user terminal device 100 by hand. The user terminal 100 recognizes that there is a user command for the copy mode through motion recognition of the user's operation using the interface unit 140 or the camera unit 160. [ The user input indicating the copy mode may be a different kind of operation, or may be a user input using a touch screen, a keypad, or the like instead of motion recognition. Referring to FIG. 19B, a screen 1910 displayed in the user terminal device 100 when a user enters a copy mode in FIG. 19A is directly copied to the display device 200, and a second screen 1910 ' And displayed. It may not display anything on the user terminal device 100 after the screen is copied to the display device 200 according to the copy mode operation. However, unlike the case shown in Fig. 19B, the execution screen of the continuous application is displayed in the user terminal device 100, and an embodiment in which only the application execution screen at the time of user input indicating the copy mode is copied to the display device 200 Can be. One of the purposes of using the copy mode is to reduce power consumption. It is possible to display the screen displayed on the user terminal device 100 on the display device 200 with low power consumption when the display device 200 with less power consumption is paired with the user terminal device 100, It is possible to reduce the amount of overall power consumption by not displaying anything to the device 100. [

The copy mode can be more useful when a static screen such as a web page or an E-book is used rather than a content in which a screen is frequently updated, such as a moving image. 20 is a diagram for explaining an embodiment in which a copy mode in an E-book is performed. The display device 200 copies the E-book screen displayed in the user terminal device 100 when a user input indicating the copy mode is made, and the second screen 2010 is configured and displayed. And the screen displayed on the user terminal device 100 is no longer displayed. For example, in the case of an e-book, since a user reads a text, a user can more easily instruct a copy mode without an additional operation by using an eye scroll, which is an eye recognition motion recognition using body parts. In another embodiment, when it is recognized that the user has completely read the E-book screen 2010 copied and displayed on the display device 200 using the camera unit 160 of the user terminal device 100, the controller 170 May immediately transmit the data corresponding to the next page to the display device 200 without displaying the screen on the user terminal device 100 again. In yet another embodiment, a function may be provided to perform a Copy Mode in conjunction with a sensor. The first screen displayed on the user terminal device 100 is copied to the second screen of the display device 200 when the screen displayed through the user terminal device 100 is shortly covered with the display device 200 Can be displayed. The control unit 170 can recognize that the operation of covering the display device with the hall sensor, the illuminance sensor, the direction / position sensor, the camera unit 160, and the like using the magnet principle has been performed. Also, when the connection angle is determined to be 0 ° by using the inter-display connection angle determination module 131-3, a method may be used in which a user command indicating the copy mode exists.

According to an embodiment of the present invention, the controller 170 displays data corresponding to a user input or a screen edit command when the display device 200 does not support user input, or when the size of the screen is small, The communication unit 120 can be controlled to transmit to the device 200. [

21 is a view for explaining an embodiment supporting touch input. Referring to FIG. 21, the display device 200 corresponds to a display device that does not support user input such as a touch. The user inputs a touch corresponding to page switching on the screen 2110 of the user terminal device 100 and the controller 170 transmits data corresponding to the user input to switch the page through the communication unit 120 to the display device 200 . The display device 200 performs an operation of switching pages on the second screen 2120 based on the received user input data. The function of controlling the display device 200 through the touch input in the user terminal device 100 as described above may be configured by a module. The display device touch input module 131-5 (not shown) that performs such a function may be included in the display attribute management module 131. [

In another embodiment, the user may perform screen editing of the display device 200 through the user terminal device 100. [ 22 is a diagram for explaining an embodiment supporting a screen editing function. 22, when the display device 200 is a device having a small screen such as a smart watch, the user terminal 100 having a larger screen can be used even if the display device 200 supports user input It is easier to edit the screen. Also, in the case of the display device 200 that does not provide a screen editing function, an embodiment that provides a screen editing function may be applied. Since it is assumed that the display device 200 selects and displays only a part of information in the IDLE screen or the home screen state, the widget information to be always displayed for each user may be different, so a method of personalizing the widget screen is required Do. A screen 2210 for editing a widget of the display device 200 paired with the home screen editing mode of the user terminal device 100 is provided and the widget corresponding to the display device 200 is edited in the user terminal device 100 The contents can be reflected on the display device 200 in real time. As another example, even when the electronic paper (E-paper) is used as the display device 200, it is possible to change the position by drag & drop when entering the E-paper Layout mode or insert / Deletion may be supported. Also, it is possible to automatically reflect the changed layout on the electronic paper (E-paper) when the Back button is input.

According to an embodiment of the present invention, when an application for photographing a picture is being executed in the user terminal device 100, the surrounding sounds are recorded together and displayed on a display device (sound-to-text) 200, a text screen may be displayed. Referring to FIG. 23, the user terminal device 100 displays a screen 2310 that is viewed through the camera unit 160 for photographing. A screen 2320 displaying the conversation contents of the two persons in text on the display device 200 is displayed by using the user classification technique according to the voice and the STT technology in the surrounding conversation during the photographing process. In addition, the text displayed on the display device 200 may be stored and provided together with a photograph file to be photographed later so that the text stored by the user may be used for grasping the atmosphere at the time of photographing. If the display type of the display device 200 is slow in update rate such as electronic paper (E-paper), it may not be possible to reflect text in real time. Accordingly, when the display device 200 is of the E-paper type, partial updating is performed only on the portion updated in the real-time conversation, thereby eliminating a problem that may occur when the entire screen is updated all at once will be.

According to another embodiment of the present invention, when the usable battery of the user terminal device 100 goes down to a certain level or lower, the display automatically switches to the display device 200 so that the screen displayed on the user terminal device 100 is displayed can do. At this time, a feature that the screen is displayed on the user terminal device 100 only when the user desires can be added. In another embodiment, in the user terminal 300 having two display units 310 and 320 as shown in FIG. 29B, the first display unit 310 is used as a main display unit, There may be an embodiment in which the screen is displayed on the second display unit 320 having a characteristic that power consumption is lower than that of the first display unit 310. [

According to an embodiment of the present invention, when the widget screen is configured in the user terminal device 100, the controller 170 controls the communication unit 120 to transmit data on the widget screen to the display device 200 . 24A, a widget screen 2420 configured in the user terminal device 100 is displayed on the display device 200, and a home screen 2410 is displayed on the user terminal device 100. FIG. In the user terminal device 100 such as a smart phone, the home screen can be scrolled left and right to have a plurality of screens, and it is possible to edit one of the screens as a widget screen. The reason for the user to configure the widget screen is to conveniently use widgets for various applications according to usage patterns for each user. Accordingly, since the widget screen 2420 composed of the applications frequently used by the user is always displayed on the display device 200, it is possible to quickly consume the content even if another user uses the home screen in the user terminal device 100 . The widget screen 2420 'may be displayed on the display device 200 even when the user terminal device 100 displays the lock screen 2410' as shown in FIG. 24B. The information update of the widget screen may be performed immediately before switching to the lock screen so as to include the latest information or may be updated each time the screen is switched to the home screen. In addition, when the case of the user terminal device 100 is switched to the home screen when the case is opened and when it is switched to the lock screen when the case is closed, the opening / closing of the case can be recognized using the hall sensor or the illuminance sensor, May be configured to update the widget information each time it is opened.

In another embodiment, even when the screen of the user terminal device 100 is turned off, the message or e-mail application receives the data. Turning on the screen of the user terminal 100 every time a message is received can cause a lot of power consumption. Accordingly, a message or e-mail information arriving from the lock screen state may be displayed in a form of a widget on the display device 200 having low power consumption. 25, since the screen of the user terminal device 100 is in the lock screen state, nothing is displayed, and the display device 200 displays a screen 2510 composed of widgets synthesizing the contents of the message. The widget screen 2510 displayed on the display device 200 is an example of applying one of various data visualization methods for displaying a plurality of data on one screen. A widget screen provided in a list according to a message reception time, and the like, and is not limited to the example shown in Fig.

The various embodiments described above are those that are implemented in the case where the connection angle between displays, which is a general case, is in the vicinity of 180 °. However, these embodiments do not necessarily have to be performed with a connection angle near 180 °. Hereinafter, embodiments in which the connection angle between the displays is in the vicinity of 270 DEG or in the vicinity of 360 DEG will be described below.

For example, the controller 170 may update the widget screen displayed on the display device 200 when it is determined that the connection angle between the displays is in the vicinity of 270 degrees. For convenience of explanation, it is assumed that the connection angle is in the 270 ° state. In the case where the connection angle is 270 °, a piezoelectric element is connected to the connection portion between the user terminal device 100 and the display device 200 So that the angle between the two devices can be determined. Alternatively, when the user terminal device 100 and the display device 200 connected to the table are wired, the angle of the user terminal device 100 in the table is detected through the direction sensor of the user terminal device 100 There may be a way to do that. In addition, a plurality of methods of grasping the connection angle between the user terminal 100 and the display device 200 may be applied.

When the connection angle between the displays is 360 °, it is important to distinguish between the general touch input and the rear touch input. The user terminal device 100 can recognize the direction of the user terminal device using the direction sensor of the user terminal device 100 and can detect the direction of the user terminal device 100 in cooperation with the camera unit 160 of the user terminal device 100. [ It is possible to determine whether the user terminal 100 is the user terminal 100 or the display device 200. The touch input when the user terminal device 100 faces the direction opposite to the direction in which the user is present is determined as the rear touch input. According to one embodiment of the present invention, when the user terminal device 100 and the display device 200 are faced with a 360 degree connection angle, and the display device 200 does not support user input, The control unit 170 recognizes that a copy mode is instructed when a rear touch input is made to the terminal device 100 so that the control unit 170 displays the screen displayed on the user terminal device 100 on the display device 200 can do.

Various embodiments are provided in which the display device 200 can be paired and used by the user terminal device 100 as described above. The user can obtain effects such as enlargement of display screen, maximization of visibility, and efficiency of power consumption.

Hereinafter, a method of controlling a user terminal will be described with reference to FIGS. 26 and 27. FIG.

26 is a flowchart illustrating a method of controlling a user terminal according to an exemplary embodiment of the present invention. First, the user terminal device executes the application and the execution screen of the application is displayed on the user terminal device (S2610). It must be paired with an external display device in order to reconstruct and display the screen being displayed in the user terminal. The user terminal determines whether pairing with an external display device has been performed (S2620). If the pairing is not performed (S2620-N), it is possible to monitor whether or not the pairing is continued. In a case where the pairing is made with the external display device (S2620-Y), the user terminal receives the information on the paired display device (S2630). The information on the display device may include at least one of a display type, a resolution, a response speed, a color, a gradation, a viewing angle, a color region, a connection angle, and a connection state of the display device. The user terminal device displays a first screen for the application based on the acquired information about the display device and information about the application running on the user terminal device and displays the data of the second screen for the application on the paired display device Lt; / RTI > In this case, the information on the application may include at least one of a category of the application, a color attribute of each part, and a screen update rate.

In one embodiment of the present invention, the execution screen of the application executed in the user terminal device is continuously displayed as the first screen, and data on the second screen including the additional information on the execution screen of the application is transmitted to the display device It is possible. In another embodiment, when the execution screen of the application includes text, data for a second screen including text or an image summarizing the contents of the included text may be transmitted to the display device. In another embodiment, when the execution screen of the application includes moving picture contents, data on the second screen including the information on the included moving picture contents or the caption may be transmitted to the display device.

In one embodiment of the present invention, when there is a user input instructing a copy mode, the user terminal device does not display the first screen, but executes the application when there is a user input instructing the copy mode The data for the screen may be transmitted to the display device. As another example, the user terminal device may continuously display the execution screen of the application as the first screen, and may transmit the data of the execution screen of the application when the user input for instructing the copy mode is made to the display device.

In another embodiment of the present invention, if the paired display device does not support user input, it may further comprise transmitting data corresponding to the user input to the display device.

27 is a detailed flowchart for explaining a control method of a user terminal according to another embodiment of the present invention. The user terminal executes the application and the execution screen of the application is displayed on the user terminal (S2710). The user terminal device is paired with an external display device so that various functions such as screen division can be provided to the user. The user terminal determines whether pairing with an external display device has been performed (S2720). If the pairing is not performed (S2720-N), the user terminal continues monitoring whether or not the pairing is performed. If the external display device is paired (S2720-Y), the user terminal receives information on the paired display device (S2730). The user terminal device divides the execution screen into a plurality of elements based on the information about the application (S2740). The user terminal device has information on the received display device and information on the application, and based on which device is maximized in visibility and power consumption efficiency when displayed, etc., each divided element is divided into a first It is determined whether the second screen becomes a constituent element of the screen (S2750). When each of the divided elements is determined as an element to be displayed on the user terminal, the first screen is displayed on the user terminal (S2760). If it is determined that the element is to be displayed on the external paired display device The user terminal device transmits the data of the second screen to the display device (S2770).

According to another embodiment, when a user terminal device divides an execution screen of an application into a plurality of elements, the user terminal device divides the application screen based on the partial screen update rate, May be configured and displayed on the user terminal device, and the second screen may be composed of the rest of the plurality of divided elements.

According to another embodiment, the user terminal device may divide the execution screen of the application into a plurality of elements based on the partial color attribute. The first screen may be composed of the elements of the first color among the plurality of divided elements, and the second screen may be composed of the elements of the second color among the plurality of elements. For example, when the display type of the user terminal is OLED and the display type of the paired display device is electronic paper (E-paper), the power consumption of white elements is large due to self-luminescence characteristics It is advantageous from the viewpoint of power efficiency that the second screen is constituted by white elements and is displayed on a display device of an E-paper type.

With the above-described control method of the user terminal, the user can experience the extended display environment and reconfigure the execution screen of the application to be more convenient to use.

Hereinafter, a sequence of the extended display system will be described with reference to FIG.

Referring to FIG. 28, the extended display system can display the first screen and the second screen in the following order. First, when an application execution screen is displayed in the user terminal device 100 (S2810), the user terminal device 100 paired with the display device 200 (S2820). In the case of being paired with the user terminal device 100, the display device 200 transmits the display device information to the user terminal device 100 (S2830). The user terminal device 100 displays the first screen for the application on the basis of the display device information transmitted from the display device 200 and the information of the executing application (S2840). In addition, the user terminal device 100 transmits the data of the second screen for the application to the display device 200 (S2850). Then, the display device 200 displays the second screen using the data of the received second screen.

Although only the case where the user terminal and the display device each having one display portion are paired is described above, the above-described embodiments can not necessarily be implemented through only two devices. For example, the present invention can be applied to a user terminal device having a plurality of display portions, and is also applicable to a case where one user terminal device and a plurality of display devices are paired. 29A is a diagram illustrating a user terminal apparatus 300 having a plurality of display units 310 and 320, and FIG. 29B is a block diagram illustrating such a user terminal apparatus 300. FIG. In the user terminal 300 having a plurality of display units, the first and second display units 310 and 320 may have different display characteristics to implement the above-described embodiments.

In another embodiment of the present invention, a display device such as an electronic paper (E-paper) may be attached to one side of the user terminal device case. 30A is a diagram showing a user terminal device case 400 including an electronic paper (E-paper). The user terminal unit case 400 may include a display unit 410, a connection unit 420, a circuit board 430, and a control unit 440. The display unit 410 may include various types of display devices. Hereinafter, for the sake of convenience of explanation, it is assumed that electronic paper (E-paper) is used. The connection unit 420 connects the user terminal unit 400 and the user terminal unit 100 to each other. The connection unit 420 may be formed of a flexible printed circuit board (FPCB) so that the display unit 410 and the user terminal device 100 are connected in a wired manner. The degree of bending of the connection portion 420 may be measured using a piezoelectric element or the like so that the user terminal device 100 may determine the connection angle with the display portion 410. The circuit board 430 may be positioned at a position where the user terminal device 100 is to be mounted and the controller 440 may be included in the circuit board 430. 30B is a view showing that the user terminal device 100 is installed in the user terminal device case 400. FIG. The various embodiments described above may be applied by using the display unit 110 of the user terminal device 100 and the display unit 410 of the user terminal device case 400 as shown in FIG. User convenience can be improved in terms of portability when the terminal 400 is used as a user terminal device case 400.

FIG. 31A is a view showing another form of a user terminal device case 500. FIG. The difference from the user terminal unit case 400 shown in Fig. 30 is that the display unit 510 is attached to the rear side and the power supply unit 550 is included. In addition, the user terminal apparatus case 500 may include a light emitting portion 560 and a light receiving portion 570. FIG. 31B is a diagram showing a state in which the user terminal device 100 is mounted on the user terminal device case 500. FIG. As shown in FIG. 31B, when the user terminal unit case 500 is opened, only the display unit 110 of the user terminal 100 displays the screen. Therefore, while the user terminal unit case 500 is open, the display unit 510 does not display a screen, and the power supply unit 550, which may be composed of a solar cell or the like, . When the user terminal unit case 500 is closed, the light emitting unit 560 comes into contact with the camera 510 of the user terminal 100 or the portion 560 'on which the optical sensor is mounted. The light receiving unit 570 is configured to correspond to the light emitting device 570 'of the user terminal device. Therefore, when the user terminal unit case 500 is closed to cover the user terminal unit 100, the user terminal unit case 500 is connected to the user terminal unit 100 using the light emitting unit 560 and the light receiving unit 570 Data can be transmitted and received, and the angle of connection can be grasped using illuminance or the like. 31C is a view showing a case where the user terminal apparatus case 500 is closed. Even when the user terminal unit case 500 is closed, the user can confirm the widget screen of the user terminal unit 100 using the display unit 510 attached to the outside of the case. Data on a screen displayed on the display unit 510 may be received by a wireless communication method using a light receiving unit 570 or an NFC tag (not shown), or may be received by a wired communication method through a connection unit 520 . The various embodiments described above can be performed through the user terminal device case 500. [ In particular, embodiments where the connection angle is 0 deg. May be applied.

In addition, the program code for performing the control method according to various embodiments as described above can be stored in various types of recording media. More specifically, it may be a random access memory (RAM), a flash memory, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an Electrically Erasable and Programmable ROM (EEPROM), a register, a hard disk, a removable disk, And may be stored in various types of recording media readable by a terminal, such as a memory, a CD-ROM, and the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

100: user terminal device 110:
120: communication unit 170:
200: display device

Claims (23)

In a user terminal,
A display unit for displaying an execution screen of the application;
A communication unit for performing communication with the display device;
Wherein when the display unit is paired with the display device through the communication unit while displaying an execution screen of the application, information about the display device is acquired, information about an application executed by the user terminal device, And a control unit for controlling the display unit to display a first screen for the application based on at least one of information on the first screen and information on the second screen and controlling the communication unit to transmit data of the second screen for the application to the display device . The method according to claim 1,
Wherein the information about the display device includes at least one of a display type, a resolution, a response speed, a color, a gradation, a viewing angle, a color area, a connection angle, and a connection state of the display device. The method according to claim 1,
Wherein the information on the application includes at least one of a category of the application, a color attribute of the application, and a screen update rate of each application. The method according to claim 1,
Wherein,
An execution screen of an application displayed on the display unit is divided into a plurality of elements based on information about the application, the first screen is configured as a part of the divided plurality of elements, And configures the second screen as a part of the second screen. 5. The method of claim 4,
Wherein,
Dividing an execution screen of an application displayed on the display unit into a plurality of elements based on a partial screen update rate, constructing the first screen with elements including a moving picture among the plurality of divided elements, Wherein the second screen comprises the remaining elements of the plurality of divided elements. 5. The method of claim 4,
Wherein,
Dividing an execution screen of an application displayed on the display unit into a plurality of elements based on a color property of each part, composing the first screen with elements of a first color among the plurality of divided elements, Wherein the second screen comprises the elements of the second color among the plurality of divided elements. The method according to claim 1,
Wherein,
Controlling the communication unit to control the display unit to continuously display an execution screen of the application as the first screen and transmit data on a second screen including additional information on the execution screen of the application to the display device To the user terminal device. 8. The method of claim 7,
Wherein,
When the execution screen of the application includes text, controls the communication unit to transmit data for a second screen including text or an image summarizing the contents of the text to the display device User terminal. 8. The method of claim 7,
Wherein,
And controls the communication unit to transmit data on a second screen including information on the included moving picture content or a caption to the display device when the execution screen of the application includes the moving picture content. Terminal device. The method according to claim 1,
Wherein,
Wherein the control unit controls the display unit not to display the first screen when there is a user input instructing a copy mode and displays data on an execution screen of the application when a user input instructing the copy mode exists Wherein the control unit controls the communication unit to transmit to the display device. The method according to claim 1,
Wherein,
Wherein the control unit controls the communication unit to transmit data corresponding to a user input to the display device when the display device does not support user input. A method of controlling a user terminal device,
Acquiring information on the display device when the user terminal device is paired with an external display device while displaying an execution screen of the application; And
Wherein the display unit displays a first screen for the application based on at least one of the information about the application executed by the user terminal and the obtained information about the display device, The method comprising the steps of: 13. The method of claim 12,
Wherein the information on the display device includes at least one of a display type, a resolution, a response speed, a color, a gradation, a viewing angle, a color region, a connection angle, and a connection state of the display device. 13. The method of claim 12,
Wherein the information on the application includes at least one of a category of the application, a color attribute of the application, and a screen update rate of each application. 13. The method of claim 12,
Dividing an execution screen of the displayed application into a plurality of elements based on information on the application; And
And constructing the first screen with a part of the plurality of divided elements and the second screen with the remaining part of the plurality of divided elements. 16. The method of claim 15,
Wherein the dividing step comprises:
Characterized in that an execution screen of the displayed application is divided into a plurality of elements based on a partial screen update rate,
Wherein the configuring comprises:
Wherein the first screen is composed of elements including moving pictures among the plurality of divided elements and the second screen is composed of remaining elements of the plurality of divided elements. 16. The method of claim 15,
Wherein the dividing step comprises:
Wherein the execution screen of the application to be displayed is divided into a plurality of elements based on the color attribute of each part,
Wherein the configuring comprises:
Wherein the first screen is constituted by the elements of the first color among the plurality of divided elements and the second screen is constituted by the elements of the second color among the plurality of elements which are divided Way. 13. The method of claim 12,
Wherein the transmitting comprises:
Wherein the control unit continuously displays the execution screen of the application as the first screen and transmits data on the second screen including the additional information on the execution screen of the application to the display device. 19. The method of claim 18,
If the execution screen of the application includes text, transmits data for a second screen including a text or an image summarizing the contents of the included text to the display device. 19. The method of claim 18,
And transmitting the data on the second screen including the information on the included moving picture content or the caption to the display device when the execution screen of the application includes the moving picture content. 13. The method of claim 12,
Wherein the transmitting comprises:
When there is a user input instructing a copy mode, data on an execution screen of the application when the user input for instructing the copy mode is not transmitted to the display device without displaying the first screen . 13. The method of claim 12,
Further comprising: if the display device does not support user input, transmitting data corresponding to the user input to the display device. A display device that is paired with the user terminal and displays data of a second screen transmitted from the user terminal; And
The information processing method according to claim 1, further comprising the steps of: acquiring information on the display device when being paged with the display device while displaying an execution screen of the application; determining, based on at least one of information on the application to be executed and information on the obtained display device And a user terminal device for displaying a first screen for the application and transmitting the data of the second screen for the application to the display device.

Images