KR20180052302A - Method for controlling source device at sink device and apparatus for the same - Google Patents

Abstract

A sink device according to an embodiment of the present invention includes a network interface part for communicating with a source device in a Wi-Fi direct mode, a display part for displaying the image of the content received from the source device, and a control part. The control part displays first application content on the first layer of the display part and displays second application content received from the source device on a second layer overlaid on the predetermined area of the first layer, and displays third application content on the first layer based on the content information transmitted from the source device. It is possible to transmit or receive information between a terminal and another device through screen mirroring.

Description

TECHNICAL FIELD [0001] The present invention relates to a sink device and a control method thereof,

The present invention relates to a display device that provides a mirroring function and a method of operating the same.

Digital TV services using wired or wireless communication networks are becoming popular. The digital TV service can provide various services that can not be provided by the existing analog broadcasting service.

For example, in the case of IPTV (Internet Protocol Television), a smart TV service, which is a kind of digital TV service, a bidirectionality is provided so that a user can actively select a type of viewing program and a viewing time. IPTV and smart TV services can provide various additional services such as Internet search, home shopping, online games and the like based on the bidirectionality.

Screen mirroring refers to a technique of directly mirroring a screen of a source device such as a mobile terminal to a screen of a sink device such as a TV using wireless communication. A user can enjoy a video or image enjoyed through a mobile terminal having a display unit of a small size as it is on a TV having a relatively large display unit by using a mirroring technique.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a sink device and a method of operating the same that allow information between a terminal and another device to be transmitted or received using a screen of another terminal provided through a screen mirroring technique.

Another object of the present invention is to minimize the operation of the mobile terminal when the screen of the mobile terminal is displayed on the display device using DLNA technology.

Another object of the present invention is to facilitate driving a plurality of applications by using an application installed in a display device in mirroring a content image of a mobile terminal.

A sink device according to an embodiment of the present invention includes a network interface unit for communicating with a source device and a Wi-Fi direct method, a display unit for displaying an image of the content received from the source device, and a control unit, Displaying a first application content on a layer and displaying a second application content received from the source device on a second layer overlaid in a certain area of the first layer, And controls the third application content to be displayed on the first layer based on the content information transmitted from the third layer.

The first layer of the sink device according to an embodiment of the present invention is located in the entire area of the display unit.

The first application in the sink device according to an embodiment of the present invention is an application installed in the sink device.

In the sink device according to an embodiment of the present invention, the content information transmitted from the source device is DLNA (Digital Living Network Alliance) content information.

In the sync device according to an embodiment of the present invention, the third application is an application installed in the sink device, and the control unit controls to output the content corresponding to the DLNA content information through the third application.

In the sink device according to an embodiment of the present invention, the content information transmitted from the source device is transmitted through a UIBC (User Input Back Channel).

The network interface unit of the sink device according to an embodiment of the present invention is characterized by receiving a coordinate table of a source device when performing a WiFi direct communication with the source device.

In the sink device according to an embodiment of the present invention, when the third application is an application installed in the source device, the control unit controls the mirroring image to be displayed on the first layer and the second layer.

A method of controlling a sink device according to an embodiment of the present invention includes performing a Wi-Fi direct communication with a source device, displaying a first application content on a first layer of a screen, Displaying a second application content received from the source device on a second layer overlaid on the region and displaying the third application content on the first layer based on the content information transmitted from the source device; The method comprising the steps of:

Further, in the method of controlling a sink device according to an embodiment of the present invention, the first layer is located in the entire area of the screen.

In the method of controlling a sink device according to an embodiment of the present invention, the first application is an application installed in a sink device.

Also, in the method of controlling a sink device according to an embodiment of the present invention, the content information transmitted from the source device is DLNA (Digital Living Network Alliance) content information.

In the method of controlling a sink device according to an embodiment of the present invention, the third application is an application installed in the sink device, and the sink device outputs the content corresponding to DLNA content information through the third application .

In the method of controlling a sink device according to an embodiment of the present invention, content information transmitted from a source device is transmitted through a user input back channel (UIBC).

In the method of controlling a sink device according to an embodiment of the present invention, performing the Wi-Fi direct communication with the source device may include receiving a coordinate table of the source device.

The effects of the present invention are as follows.

According to an embodiment of the present invention, information between a terminal and another apparatus can be easily transmitted or received using a screen of another terminal provided through a screen mirroring technology.

According to another embodiment of the present invention, the operation of the mobile terminal is minimized when the screen of the mobile terminal is displayed on the display device using the DLNA technology.

According to another embodiment of the various embodiments of the present invention, there is an effect of facilitating the driving of a plurality of applications by using the application installed in the display device in mirroring the content image of the mobile terminal.

1 is a block diagram illustrating a configuration of a display apparatus according to an embodiment of the present invention.
2 is a block diagram of a remote control device according to an embodiment of the present invention.
3 shows an example of an actual configuration of a remote control apparatus according to an embodiment of the present invention.
4 shows an example of utilizing a remote control device according to an embodiment of the present invention.
5 is a block diagram illustrating a mobile terminal according to the present invention.
6 is a ladder diagram illustrating a process of providing a screen sharing service by establishing a WFD session between a WFD source and a WFD sink according to an embodiment of the present invention.
7 to 9 are diagrams for explaining a table showing resolution information transmitted by a WFD sink to a WFD source according to an embodiment of the present invention.
10 is a flowchart for explaining a UIBC connection between a sink device and a source device.
11 is a flowchart showing an embodiment of a method of operating a source device according to an embodiment of the present invention.
12 and 13 are diagrams for explaining a method of controlling a source device and a sink device according to an embodiment of the present invention.
14 is a flowchart for explaining another example of a method for controlling data transmitted from a source device to a sink device according to an embodiment of the present invention.
FIG. 15 is a diagram illustrating an example in which a screen displayed on a mobile terminal is displayed on a TV using a DLNA function.
16 and 17 are views showing an example of mode switching of a mobile terminal when a mobile terminal and a TV are connected by DLNA in the present invention.
18 is a diagram illustrating the entire operation of DLNA web browsing through a mobile terminal and a TV in the present invention.
19 is a flowchart illustrating an operation method of a terminal according to an embodiment of the present invention.
20 is a diagram for explaining an embodiment of outputting an execution screen of a first program as a whole screen according to an embodiment of the present invention.
21 is a view for explaining an embodiment in which a second window is additionally output by dividing the entire screen into a first area and a second area according to an embodiment of the present invention.
22 to 23 show examples of screens provided when one or more program items are selected in the program list on the second window according to an embodiment of the present invention.
24 to 26 are views for explaining an embodiment for additionally outputting an execution screen of another program through a second window for providing a multitasking environment according to an embodiment of the present invention.
27 to 28 are views for explaining an embodiment in which the layout constituting the entire screen is changed according to another embodiment of the present invention, and multitasking work can be performed through the changed layout.
29 to 30 show an example of a screen provided when an external input icon for screen mirroring is selected from an external input list on the second window according to an embodiment of the present invention.
31 to 32 show an example of a screen provided when an external input icon representing an HDMI input is selected from an external input list on the second window according to an embodiment of the present invention.
FIG. 33 shows an example of a screen provided when an external input icon indicating a TV input among the external input list on the second window is selected according to an embodiment of the present invention.
34 to 37 are diagrams for explaining an example in which content is executed through a DLNA during mirroring between a source device and a sink device according to an embodiment of the present invention.
38 is a view for explaining an example of executing content through Deep Linking during mirroring between a source device and a sink device according to an embodiment of the present invention.
FIG. 39 is a view for explaining an example of displaying a mirroring icon when mirroring is stopped between a source device and a sink device according to an embodiment of the present invention. FIG.

Hereinafter, embodiments related to the present invention will be described in detail with reference to the drawings. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role.

The display device according to the embodiment of the present invention is an intelligent display device that adds a computer support function to a broadcast reception function, for example, and is equipped with a function of inputting a handwriting, a touch screen Or an interface that is more convenient to use than a space remote controller or the like. Also, it can be connected to the Internet and a computer by the support of a wired or wireless Internet function, and can perform functions such as e-mail, web browsing, banking or game. A standardized general-purpose OS can be used for these various functions.

Therefore, the display device described in the present invention can freely add or delete various applications, for example, on a general-purpose OS kernel, so that various user-friendly functions can be performed. More specifically, the display device may be, for example, a network TV, an HBBTV, a smart TV, an LED TV, an OLED TV, or the like, and may be applied to a smartphone as the case may be.

Hereinafter, a source device and a mobile terminal can be used in combination of a sink device and a display device.

1 is a block diagram illustrating a configuration of a display apparatus according to an embodiment of the present invention.

1, the display device 100 includes a broadcast receiver 130, an external device interface 135, a storage 140, a user input interface 150, a controller 170, a wireless communication unit 173, A display unit 180, an audio output unit 185, and a power supply unit 590.

The broadcast receiving unit 130 may include a tuner 131, a demodulation unit 132, and a network interface unit 133.

The tuner 131 can tune to a specific broadcast channel according to the channel tuning command. The tuner 131 may receive a broadcast signal for the selected broadcast channel.

The demodulator 132 can separate the received broadcast signal into a video signal, an audio signal, and a data signal related to a broadcast program, and can restore the separated video signal, audio signal, and data signal into a form capable of outputting.

The external device interface unit 135 may receive an application or application list in an adjacent external device and may transmit the received application or application list to the control unit 170 or the storage unit 140.

The external device interface unit 135 may provide a connection path between the display device 100 and an external device. The external device interface unit 135 may receive at least one of video and audio output from an external device connected to the display device 100 wirelessly or by wire, and may transmit the received video and audio to the controller 170. The external device interface unit 135 may include a plurality of external input terminals. The plurality of external input terminals may include an RGB terminal, one or more HDMI (High Definition Multimedia Interface) terminals, and a component terminal.

The video signal of the external device input through the external device interface unit 135 may be output through the display unit 180. [ The audio signal of the external device input through the external device interface unit 135 may be output through the audio output unit 185. [

The external device that can be connected to the external device interface unit 135 may be any one of a set top box, a Blu-ray player, a DVD player, a game machine, a sound bar, a smart phone, a PC, a USB memory and a home theater, .

The network interface unit 133 may provide an interface for connecting the display device 100 to a wired / wireless network including the Internet network. The network interface unit 133 can transmit or receive data to another user or another electronic device via the network connected thereto or another network linked to the connected network.

In addition, some content data stored in the display device 100 can be transmitted to another user registered in the display device 100 or a selected one of the other electronic devices or selected electronic devices.

The network interface unit 133 can access a predetermined web page through the connected network or another network linked to the connected network. That is, it is possible to access a predetermined web page through a network and transmit or receive data with the server.

The network interface unit 133 may receive content or data provided by the content provider or the network operator. That is, the network interface unit 133 can receive contents such as a movie, an advertisement, a game, a VOD, a broadcast signal, and related information provided from a content provider or a network provider through a network.

In addition, the network interface unit 133 can receive update information and an update file of the firmware provided by the network operator, and can transmit data to the Internet or a content provider or a network operator.

The network interface unit 133 can select and receive a desired application among applications open to the public via the network.

The storage unit 140 stores programs for signal processing and control in the controller 170, and can store image-processed video, audio, or data signals.

The storage unit 140 may perform a function for temporarily storing video, audio, or data signals input from the external device interface unit 135 or the network interface unit 133, It may also store information about the image.

The storage unit 140 may store a list of applications or applications input from the external device interface unit 135 or the network interface unit 133. [

The display apparatus 100 may reproduce and provide a content file (a moving picture file, a still picture file, a music file, a document file, an application file, etc.) stored in the storage unit 140 to a user.

The user input interface unit 150 may transmit a signal input by the user to the control unit 170 or may transmit a signal from the control unit 170 to the user. For example, the user input interface unit 150 may be implemented by various communication methods such as Bluetooth, Ultra Wideband (WB), ZigBee, RF (Radio Frequency) communication, and infrared (IR) It is possible to receive and process control signals such as power on / off, channel selection, screen setting, and the like from the remote control device 200, or transmit control signals from the control unit 170 to the remote control device 200. [

The user input interface unit 150 may transmit a control signal input from a local key (not shown) such as a power key, a channel key, a volume key, and a set value to the controller 170.

The video signal processed by the controller 170 may be input to the display unit 180 and displayed as an image corresponding to the video signal. The video signal processed by the controller 170 may be input to the external output device through the external device interface 135. [

The audio signal processed by the control unit 170 may be output to the audio output unit 185 through audio. The voice signal processed by the control unit 170 may be input to the external output device through the external device interface unit 135.

In addition, the control unit 170 can control the overall operation in the display device 100. [

The control unit 170 may control the display device 100 according to a user command or an internal program input through the user input interface unit 150. The control unit 170 accesses the network and displays a list of applications or applications desired by the user, So that it can be downloaded into the portable terminal 100.

The control unit 170 can output the video or audio signal processed through the channel information selected by the user through the display unit 180 or the audio output unit 185.

The control unit 170 controls the operation of the external device through the external device interface unit 135 according to an external device video playback command received through the user input interface unit 150. For example, So that a video signal or a voice signal can be output through the display unit 180 or the audio output unit 185.

The control unit 170 may control the display unit 180 to display an image and may output the broadcast image input through the tuner 131 or the external input through the external device interface unit 135, The image input through the network interface unit or the image stored in the storage unit 140 may be displayed on the display unit 180. [ In this case, the image displayed on the display unit 180 may be a still image or a moving image, and may be a 2D image or a 3D image.

In addition, the controller 170 may control the content stored in the display device 100, the received broadcast content, or external input content input from the outside to be reproduced, and the content may be a broadcast image, an external input image, , A still image, a connected web screen, and a document file.

The wireless communication unit 173 can perform communication with an external device through wired or wireless communication. The wireless communication unit 173 may perform short range communication with an external device. To this end, the wireless communication unit 173 may be a wireless communication unit such as a Bluetooth ™, a Radio Frequency Identification (RFID), an Infrared Data Association (IrDA), a UWB (Ultra Wideband), a ZigBee, (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The wireless communication unit 173 may communicate with the display device 100 and the wireless communication system through the wireless area networks or between the display device 100 and the other display device 100 or between the display device 100 and the display device 100. [ And can support wireless communication between networks in which the display device 100 (or an external server) is located. The short-range wireless communication network may be a short-range wireless personal area network.

Here, the other display device 100 is a wearable device (e.g., a smartwatch, smart glass, etc.) capable of interchanging data with the display device 100 according to the present invention a smart glass, an HMD (head mounted display), or a smart phone. The wireless communication unit 173 can detect (or recognize) a wearable device that can communicate with the display device 100. When the detected wearable device is a device authenticated to communicate with the display device 100 according to the present invention, the control unit 170 controls the wireless communication unit 173 to transmit at least a part of the data processed in the display device 100 To the wearable device. Therefore, the user of the wearable device can use the data processed in the display device 100 through the wearable device.

The display unit 180 converts the video signal, the data signal, the OSD signal, the video signal, the data signal, and the like received from the external device interface unit 135 into R, G, and B signals, respectively, Signal can be generated.

On the other hand, the display device 100 shown in FIG. 1 is only an embodiment of the present invention. Some of the components shown may be integrated, added, or omitted depending on the specifications of the display device 100 actually implemented.

That is, two or more constituent elements may be combined into one constituent element, or one constituent element may be constituted by two or more constituent elements, if necessary. In addition, the functions performed in each block are intended to illustrate the embodiments of the present invention, and the specific operations and apparatuses do not limit the scope of the present invention.

1, the display apparatus 100 may include a tuner 131 and a demodulator 132 and may include a network interface unit 133 or an external device interface unit (not shown) 135 to reproduce the image.

For example, the display apparatus 100 may be divided into an image processing apparatus such as a set-top box for receiving broadcast signals or contents according to various network services, and a content reproducing apparatus for reproducing contents inputted from the image processing apparatus Can be implemented.

In this case, the operation method of the display device according to the embodiment of the present invention to be described below may be applied not only to the display device 100 as described with reference to FIG. 1, but also to the image processing device such as the separated set- 180, and an audio output unit 185. The content playback apparatus may be implemented by any one of the above-described content playback apparatuses.

Next, a remote control apparatus according to an embodiment of the present invention will be described with reference to Figs. 2 to 3. Fig.

FIG. 2 is a block diagram of a remote control apparatus according to an embodiment of the present invention, and FIG. 3 shows an example of an actual configuration of a remote control apparatus 200 according to an embodiment of the present invention.

2, the remote control apparatus 200 includes a fingerprint recognition unit 210, a wireless communication unit 220, a user input unit 230, a sensor unit 240, an output unit 250, a power supply unit 260 A storage unit 270, a control unit 280, and a voice acquiring unit 290. [0034]

Referring to FIG. 2, the wireless communication unit 225 transmits / receives a signal to / from any one of the display devices according to the embodiments of the present invention described above.

The remote control device 200 includes an RF module 221 capable of transmitting and receiving signals to and from the display device 100 according to the RF communication standard and is capable of transmitting and receiving signals with the display device 100 according to the IR communication standard And an IR module 223. In addition, the remote control device 200 may include a Bluetooth module 225 capable of transmitting and receiving signals with the display device 100 according to the Bluetooth communication standard. The remote control apparatus 200 is provided with an NFC module 227 capable of transmitting and receiving signals to and from the display apparatus 100 in accordance with a NFC (Near Field Communication) communication standard. In accordance with a WLAN (Wireless LAN) communication standard, And a WLAN module 229 that can send and receive signals to and from the device 100.

The remote control device 200 transmits a signal containing information on the motion and the like of the remote control device 200 to the display device 100 through the wireless communication unit 220.

On the other hand, the remote control device 200 can receive a signal transmitted from the display device 100 through the RF module 221 and can transmit power to the display device 100 via the IR module 223, Off, channel change, volume change, and the like.

The user input unit 230 may include a keypad, a button, a touch pad, or a touch screen. The user can input a command related to the display device 100 to the remote control device 200 by operating the user input unit 230. [ When the user input unit 230 has a hard key button, the user can input a command related to the display device 100 to the remote control device 200 through the push operation of the hard key button. This will be described with reference to FIG.

Referring to FIG. 3, the remote control apparatus 200 may include a plurality of buttons. The plurality of buttons include a fingerprint recognition button 212, a power button 231, a home button 232, a live button 233, an external input button 234, a volume control button 235, a voice recognition button 236, A channel change button 237, an OK button 238, and a back button 239. [

The fingerprint recognition button 212 may be a button for recognizing the fingerprint of the user. In one embodiment, the fingerprint recognition button 212 is capable of a push operation and may receive a push operation and a fingerprint recognition operation. The power button 231 may be a button for turning on / off the power of the display device 100. The home button 232 may be a button for moving to the home screen of the display device 100. The live button 233 may be a button for displaying a real time broadcast program. The external input button 234 may be a button for receiving an external input connected to the display device 100. The volume control button 235 may be a button for controlling the volume of the volume output by the display device 100. The voice recognition button 236 may be a button for receiving the user's voice and recognizing the received voice. The channel change button 237 may be a button for receiving a broadcast signal of a specific broadcast channel. The confirmation button 238 may be a button for selecting a specific function, and the back button 239 may be a button for returning to a previous screen.

Fig. 2 will be described again.

When the user input unit 230 has a touch screen, the user can touch a soft key of the touch screen to input a command related to the display device 100 to the remote control device 200. [ In addition, the user input unit 230 may include various types of input means such as a scroll key, a jog key, and the like that can be operated by the user, and the present invention does not limit the scope of the present invention.

The sensor unit 240 may include a gyro sensor 241 or an acceleration sensor 243 and the gyro sensor 241 may sense information on the motion of the remote control device 200. [

For example, the gyro sensor 241 may sense information about the operation of the remote control device 200 on the basis of the x, y, and z axes, and the acceleration sensor 243 may sense the movement speed of the remote control device 200 And the like can be sensed. The remote control device 200 may further include a distance measuring sensor to sense the distance from the display unit 180 of the display device 100. [

The output unit 250 may output an image or voice signal corresponding to the operation of the user input unit 235 or corresponding to the signal transmitted from the display device 100. [ The user can recognize whether the user input unit 235 is operated or whether the display apparatus 100 is controlled through the output unit 250.

For example, the output unit 250 may include an LED module 251 that is turned on when the user input unit 235 is operated or a signal is transmitted to and received from the display device 100 through the wireless communication unit 225, a vibration module 253 for outputting sound, an acoustic output module 255 for outputting sound, or a display module 257 for outputting an image.

In addition, the power supply unit 260 supplies power to the remote control device 200, and if the remote control device 200 is not moved for a predetermined time, the power supply is interrupted to reduce power consumption. The power supply unit 260 may resume power supply when a predetermined key provided in the remote control device 200 is operated.

The storage unit 270 may store various types of programs, application data, and the like necessary for the control or operation of the remote control apparatus 200. [ If the remote control device 200 wirelessly transmits and receives signals through the display device 100 and the RF module 221, the remote control device 200 and the display device 100 transmit and receive signals through a predetermined frequency band .

The control unit 280 of the remote control device 200 stores information on a frequency band and the like capable of wirelessly transmitting and receiving a signal with the display device 100 paired with the remote control device 200 in the storage unit 270, can do.

The control unit 280 controls various matters related to the control of the remote control device 200. [ The control unit 280 transmits a signal corresponding to a predetermined key operation of the user input unit 235 or a signal corresponding to the motion of the remote control device 200 sensed by the sensor unit 240 through the wireless communication unit 225 100).

Also, the voice acquisition unit 290 of the remote control device 200 can acquire voice.

The voice acquisition unit 290 may include at least one microphone 291 and may acquire voice through the microphone 291.

Next, Fig. 4 will be described.

4 shows an example of utilizing a remote control device according to an embodiment of the present invention.

4A illustrates that the pointer 205 corresponding to the remote control device 200 is displayed on the display unit 180. In FIG.

The user can move or rotate the remote control device 200 up, down, left or right. The pointer 205 displayed on the display unit 180 of the display device 100 corresponds to the movement of the remote control device 200. [ In this remote control device 200, as shown in the figure, since the pointer 205 is moved according to the movement in the 3D space, it can be called a space remote controller.

4B illustrates that when the user moves the remote controller 200 to the left, the pointer 205 displayed on the display unit 180 of the display apparatus 100 moves to the left correspondingly.

Information on the motion of the remote control device 200 sensed through the sensor of the remote control device 200 is transmitted to the display device 100. The display device 100 can calculate the coordinates of the pointer 205 from the information about the motion of the remote control device 200. [ The display device 100 can display the pointer 205 so as to correspond to the calculated coordinates.

4C illustrates a case where the user moves the remote control device 200 away from the display unit 180 while the specific button in the remote control device 200 is depressed. Thereby, the selected area in the display unit 180 corresponding to the pointer 205 can be zoomed in and displayed.

Conversely, when the user moves the remote control apparatus 200 close to the display unit 180, the selection area within the display unit 180 corresponding to the pointer 205 may be zoomed out and displayed.

On the other hand, when the remote control device 200 moves away from the display unit 180, the selected area is zoomed out, and when the remote control device 200 approaches the display unit 180, the selected area may be zoomed in.

Further, the recognition of the upward, downward, and leftward / rightward movement can be excluded when a specific button in the remote control device 200 is pressed. That is, when the remote control device 200 moves away from or approaches the display unit 180, it is not recognized that the up, down, left, and right movements are recognized, and only the forward and backward movements are recognized. Only the pointer 205 is moved in accordance with the upward, downward, leftward, and rightward movement of the remote control device 200 in a state where the specific button in the remote control device 200 is not pressed.

On the other hand, the moving speed and moving direction of the pointer 205 may correspond to the moving speed and moving direction of the remote control device 200.

The pointer in the present specification means an object displayed on the display unit 180 in correspondence with the operation of the remote control device 200. [ Therefore, various shapes of objects other than the arrow shapes shown in the drawings are possible with the pointer 205. [ For example, it can be a concept that includes points, cursors, prompts, thick outlines, and so on. The pointer 205 may be displayed corresponding to a point on a horizontal axis or a vertical axis on the display unit 180 or may be displayed corresponding to a plurality of points such as a line or a surface Do.

Next, the configuration of the mobile terminal according to the present invention will be described with reference to Fig.

5 is a block diagram illustrating a mobile terminal according to the present invention.

The mobile terminal 500 includes a wireless communication unit 510, an input unit 520, a sensing unit 540, an output unit 550, an interface unit 560, a memory 570, a control unit 580, and a power supply unit 590, And the like. The components shown in FIG. 5 are not essential for implementing a mobile terminal, so that the mobile terminal described herein may have more or fewer components than the components listed above.

The wireless communication unit 510 may be connected between the mobile terminal 500 and the wireless communication system or between the mobile terminal 500 and another mobile terminal 500 or between the mobile terminal 500 and the external server 500. [ Lt; RTI ID = 0.0 > wireless < / RTI > In addition, the wireless communication unit 510 may include one or more modules for connecting the mobile terminal 500 to one or more networks.

The wireless communication unit 510 may include at least one of a broadcast receiving module 511, a mobile communication module 512, a wireless Internet module 513, a short distance communication module 514, and a location information module 515.

The input unit 520 includes a camera 521 or an image input unit for inputting an image signal, a microphone 522 for inputting an audio signal or an audio input unit, a user input unit 523 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 520 may be analyzed and processed by a user's control command.

The sensing unit 540 may include at least one sensor for sensing at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information. For example, the sensing unit 540 may include a proximity sensor 541, an illumination sensor 542, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 226, a battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification can combine and utilize information sensed by at least two of the sensors.

The output unit 550 includes at least one of a display unit 551, an acoustic output unit 552, a haptic module 553, and a light output unit 554 to generate an output related to visual, auditory, can do. The display unit 551 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. Such a touch screen may function as a user input 523 that provides an input interface between the mobile terminal 500 and a user and may provide an output interface between the mobile terminal 500 and the user.

The interface unit 560 serves as a path to various kinds of external devices connected to the mobile terminal 500. The interface unit 560 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the mobile terminal 500, corresponding to the connection of the external device to the interface unit 560, it is possible to perform appropriate control related to the connected external device.

In addition, the memory 570 stores data supporting various functions of the mobile terminal 500. The memory 570 may store a plurality of application programs or applications running on the mobile terminal 500, data for operation of the mobile terminal 500, and commands. At least some of these applications may be downloaded from an external server via wireless communication. At least some of these applications may reside on the mobile terminal 500 from the time of departure for the basic functions of the mobile terminal 500 (e.g., call incoming, outgoing, message receiving, originating). Meanwhile, the application program may be stored in the memory 570, installed on the mobile terminal 500, and operated by the control unit 580 to perform the operation (or function) of the mobile terminal.

The control unit 580 typically controls the overall operation of the mobile terminal 500, in addition to the operations associated with the application program. The control unit 580 processes signals, data, information, and the like input or output through the above-mentioned components, or drives an application program stored in the memory 570 to provide or process appropriate information or functions to the user.

In addition, the control unit 580 may control at least some of the components illustrated in FIG. 5 to drive an application program stored in the memory 570. FIG. In addition, the controller 580 may operate at least two of the components included in the mobile terminal 500 in combination with each other for driving the application program.

Under the control of the controller 580, the power supply unit 590 receives external power and internal power, and supplies power to the components included in the mobile terminal 500. The power supply unit 590 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with one another to implement a method of operation, control, or control of a mobile terminal according to various embodiments described below. In addition, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 570. [

Hereinafter, the components listed above will be described in more detail with reference to FIG. 5 before explaining various embodiments implemented through the mobile terminal 500 as described above.

First, referring to the wireless communication unit 510, the broadcast receiving module 511 of the wireless communication unit 510 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast receiving modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The mobile communication module 512 may be implemented in a mobile communication system such as a mobile communication system or a wireless communication system using a communication system such as a communication system (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution And an external terminal, or a server on a mobile communication network established according to a long term evolution (AR), a long term evolution (AR), or the like.

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 513 is a module for wireless Internet access, and may be embedded in the mobile terminal 500 or externally. The wireless Internet module 513 is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 513 transmit and receive data according to at least one wireless Internet technology, including Internet technologies not listed above.

The wireless Internet module 513 for performing a wireless Internet connection through the mobile communication network may be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE or LTE- May be understood as a kind of the mobile communication module 512.

The short-range communication module 514 is for short-range communication and may be a Bluetooth ™, a Radio Frequency Identification (RFID), an Infrared Data Association (IrDA), a UWB (Ultra Wideband), a ZigBee, (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short-range communication module 514 may communicate with the mobile terminal 500 and the wireless communication system through the wireless area networks, between the mobile terminal 500 and another mobile terminal 500, ) And the other mobile terminal 100 (or the external server). The short-range wireless communication network may be a short-range wireless personal area network.

Here, the other mobile terminal 500 may be a wearable device (e.g., a smartwatch, a smart glass, etc.) capable of interchanging data with the mobile terminal 500 according to the present invention (smart glass), HMD (head mounted display)). The short range communication module 514 can detect (or recognize) a wearable device capable of communicating with the mobile terminal 500 around the mobile terminal 500. If the detected wearable device is a device authenticated to communicate with the mobile terminal 500 according to the present invention, the control unit 580 may transmit at least part of the data processed by the mobile terminal 500 to the short- 514 to the wearable device. Therefore, the user of the wearable device can use the data processed by the mobile terminal 500 through the wearable device. For example, according to this, when a telephone is received in the mobile terminal 500, the user performs a telephone conversation via the wearable device, or when a message is received in the mobile terminal 500, It is possible to check the message.

The position information module 515 is a module for obtaining the position (or current position) of the mobile terminal, and a representative example thereof is a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the mobile terminal utilizes the GPS module, it can acquire the position of the mobile terminal by using a signal transmitted from the GPS satellite. As another example, when the mobile terminal utilizes the Wi-Fi module, it can acquire the position of the mobile terminal based on information of a wireless access point (AP) that transmits or receives the wireless signal with the Wi-Fi module. Optionally, the location information module 115 may perform any of the other functions of the wireless communication unit 510 to obtain data regarding the location of the mobile terminal, in addition or in addition. The location information module 115 is a module used to obtain the location (or current location) of the mobile terminal, and is not limited to a module that directly calculates or obtains the location of the mobile terminal.

The input unit 520 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, Or a plurality of cameras 521. The camera 521 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 551 or stored in the memory 570. [ A plurality of cameras 521 provided in the mobile terminal 500 may be arranged to have a matrix structure and various angles or foci may be provided to the mobile terminal 500 through the camera 521 having a matrix structure A plurality of pieces of image information can be input. In addition, the plurality of cameras 521 may be arranged in a stereo structure to acquire a left image and a right image for realizing the stereoscopic image.

The microphone 522 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or an executing application program) being executed in the mobile terminal 500. Meanwhile, various noise elimination algorithms may be implemented in the microphone 522 to remove noise generated in receiving an external sound signal.

The user input unit 523 is for receiving information from the user. When information is inputted through the user input unit 523, the control unit 580 can control the operation of the mobile terminal 500 to correspond to the input information . The user input unit 523 may include a mechanical input means (or a mechanical key such as a button located on the rear or side of the mobile terminal 500, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, And a touch key disposed on the touch panel. Meanwhile, the virtual key or the visual key can be displayed on a touch screen having various forms, for example, a graphic, a text, an icon, a video, As shown in FIG.

Meanwhile, the sensing unit 540 senses at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information, and generates a corresponding sensing signal. The control unit 580 may control the driving or operation of the mobile terminal 500 or may perform data processing, function or operation related to the application program installed in the mobile terminal 500 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 540 will be described in more detail.

First, the proximity sensor 541 refers to a sensor that detects the presence of an object approaching a predetermined detection surface or an object existing in the vicinity of the detection surface without mechanical contact using an electromagnetic force or an infrared ray. The proximity sensor 541 may be disposed in the inner area of the mobile terminal or the proximity sensor 541 near the touch screen.

Examples of the proximity sensor 541 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 541 can be configured to detect the proximity of the object with a change of the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as "proximity touch & The act of actually touching an object on the screen is called a "contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 541 can detect proximity touches and proximity touch patterns (e.g., proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch location, proximity touch movement state, have. Meanwhile, the control unit 580 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 541 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Further, the control unit 580 can control the mobile terminal 500 such that different operations or data (or information) are processed depending on whether a touch to the same point on the touch screen is a proximity touch or a contact touch .

The touch sensor uses a touch (or touch input) applied to the touch screen (or the display unit 551) by using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, Detection.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and then transmits the corresponding data to the controller 580. Thus, the control unit 580 can know which area of the display unit 551 is touched or the like. Here, the touch controller may be a separate component from the control unit 580, and may be the control unit 580 itself.

On the other hand, the control unit 580 may perform different controls or perform the same control according to the type of the touch object touching the touch screen (or a touch key provided on the touch screen). Whether to perform different controls or to perform the same control according to the type of the touch object may be determined according to the current state of the mobile terminal 500 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. On the other hand, the controller 580 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 521 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 521 and the laser sensor can be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 551 displays (outputs) information to be processed by the mobile terminal 500. For example, the display unit 551 may display execution screen information of an application program driven by the mobile terminal 500 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

Also, the display unit 551 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.

The audio output unit 552 can output audio data received from the wireless communication unit 510 or stored in the memory 570 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 552 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed by the mobile terminal 500. [ The sound output unit 552 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 553 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 553 may be vibration. The intensity and pattern of the vibration generated in the haptic module 553 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 553 may synthesize and output different vibrations or sequentially output the vibrations.

In addition to the vibration, the haptic module 553 may include a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, contact with an electrode, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 553 can transmit a tactile effect through direct contact, and can be implemented so that a user can feel the tactile effect through a muscular sense such as a finger or an arm. The haptic module 553 may include two or more haptic modules according to the configuration of the mobile terminal 500.

The light output unit 554 outputs a signal for notifying the occurrence of an event using the light of the light source of the mobile terminal 500. Examples of events that occur in the mobile terminal 500 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output by the light output unit 554 is implemented as the mobile terminal emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the mobile terminal detecting the event confirmation of the user.

The interface unit 560 serves as a path for communication with all external devices connected to the mobile terminal 500. The interface unit 560 receives data from an external device or supplies power to each component in the mobile terminal 500 or allows data in the mobile terminal 500 to be transmitted to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 560.

The identification module is a chip for storing various information for authenticating the use right of the mobile terminal 500 and includes a user identification module (UIM), a subscriber identity module (SIM) A universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the interface unit 560. [

The interface unit 560 may be a path through which power from the cradle is supplied to the mobile terminal 500 when the mobile terminal 500 is connected to an external cradle, And various command signals may be transmitted to the mobile terminal 500. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal 500 is correctly mounted on the cradle.

The memory 570 may store a program for the operation of the control unit 580 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 570 may store data on vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 570 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The mobile terminal 500 may be operated in association with a web storage that performs the storage function of the memory 570 on the Internet.

Meanwhile, as described above, the control unit 580 controls an operation related to an application program and an overall operation of the mobile terminal 500. [ For example, when the state of the mobile terminal meets a set condition, the controller 580 may execute or release a lock state for restricting input of a user's control command to applications.

In addition, the control unit 580 performs control and processing related to voice communication, data communication, video call, and the like, or performs pattern recognition processing that can recognize handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the controller 580 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the mobile terminal 500 according to the present invention.

The power supply unit 590 receives external power and internal power under the control of the controller 580 and supplies power required for operation of the respective components. The power supply unit 590 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 590 may include a connection port, and the connection port may be configured as an example of the interface 160 in which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 590 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 590 may use at least one of an inductive coupling method based on a magnetic induction phenomenon and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon from an external wireless power transmission apparatus Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

Next, a communication system that can be implemented through the mobile terminal 500 according to the present invention will be described.

First, the communication system may use different wireless interfaces and / or physical layers. For example, wireless interfaces that can be used by a communication system include Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA) ), Universal mobile telecommunication systems (UMTS) (in particular Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A)), Global System for Mobile Communications May be included.

Hereinafter, for the sake of convenience of description, the description will be limited to CDMA. However, it is apparent that the present invention can be applied to all communication systems including an OFDM (Orthogonal Frequency Division Multiplexing) wireless communication system as well as a CDMA wireless communication system.

A CDMA wireless communication system includes at least one terminal 100, at least one base station (BS) (also referred to as a Node B or Evolved Node B), at least one Base Station Controllers (BSCs) , And a Mobile Switching Center (MSC). The MSC is configured to be coupled to a Public Switched Telephone Network (PSTN) and BSCs. The BSCs may be paired with the BS via a backhaul line. The backhaul line may be provided according to at least one of E1 / T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL or xDSL. Thus, a plurality of BSCs may be included in a CDMA wireless communication system.

Each of the plurality of BSs may comprise at least one sector, and each sector may comprise an omnidirectional antenna or an antenna pointing to a particular direction of radial emission from the BS. In addition, each sector may include two or more antennas of various types. Each BS may be configured to support a plurality of frequency assignments, and a plurality of frequency assignments may each have a specific spectrum (e.g., 1.25 MHz, 5 MHz, etc.).

The intersection of sector and frequency assignment may be referred to as a CDMA channel. The BS may be referred to as a base station transceiver subsystem (BTSs). In this case, a combination of one BSC and at least one BS may be referred to as a "base station ". The base station may also indicate a "cell site ". Alternatively, each of the plurality of sectors for a particular BS may be referred to as a plurality of cell sites.

A broadcast transmission unit (BT) transmits a broadcast signal to terminals 100 operating in the system. The broadcast receiving module 511 shown in FIG. 5 is provided in the terminal 100 to receive a broadcast signal transmitted by the BT.

In addition, a Global Positioning System (GPS) may be associated with the CDMA wireless communication system to identify the location of the mobile terminal 500. The satellite 300 aids in locating the mobile terminal 500. Useful location information may be obtained by two or more satellites. Here, the position of the mobile terminal 500 can be tracked using all the techniques capable of tracking the location as well as the GPS tracking technology. Also, at least one of the GPS satellites may optionally or additionally be responsible for satellite DMB transmissions.

The location information module 515 included in the mobile terminal is for detecting, computing, or identifying the location of the mobile terminal, and may include a Global Position System (GPS) module and a WiFi (Wireless Fidelity) module. Optionally, the location information module 115 may perform any of the other functions of the wireless communication unit 510 to obtain data regarding the location of the mobile terminal, in addition or in addition.

The GPS module 115 calculates distance information and accurate time information from three or more satellites and then applies trigonometry to the calculated information to accurately calculate three-dimensional current location information according to latitude, longitude, and altitude can do. At present, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another satellite is widely used. In addition, the GPS module 115 can calculate speed information by continuously calculating the current position in real time. However, it is difficult to accurately measure the position of the mobile terminal by using the GPS module in the shadow area of the satellite signal as in the room. Accordingly, a WPS (WiFi Positioning System) can be utilized to compensate the positioning of the GPS system.

The WiFi Positioning System (WPS) is a system in which a mobile terminal 500 is controlled by using a WiFi module provided in the mobile terminal 500 and a wireless AP (wireless access point) transmitting or receiving a wireless signal with the WiFi module. Is a technology for tracking a location of a wireless local area network (WLAN) using WiFi.

The WiFi location tracking system may include a Wi-Fi location server, a mobile terminal 500, a wireless AP connected to the mobile terminal 500, and a database in which certain wireless AP information is stored.

The mobile terminal 500 connected to the wireless AP can transmit the location information request message to the Wi-Fi location server.

The Wi-Fi location server extracts information of the wireless AP connected to the mobile terminal 500 based on the location information request message (or signal) of the mobile terminal 500. The information of the wireless AP connected to the mobile terminal 500 may be transmitted to the Wi-Fi location server through the mobile terminal 500 or may be transmitted from the wireless AP to the Wi-Fi location server.

The information of the wireless AP to be extracted based on the location information request message of the mobile terminal 500 includes a MAC Address, an SSID (Service Set IDentification), a Received Signal Strength Indicator (RSSI), a Reference Signal Received Power (RSRP) Reference Signal Received Quality), channel information, Privacy, Network Type, Signal Strength, and Noise Strength.

As described above, the Wi-Fi position location server can receive the information of the wireless AP connected to the mobile terminal 500 and extract the wireless AP information corresponding to the wireless AP to which the mobile terminal is connected from the pre-established database. In this case, the information of any wireless APs stored in the database includes at least one of MAC address, SSID, channel information, privacy, network type, radius coordinates of the wireless AP, building name, Available), the address of the AP owner, telephone number, and the like. At this time, in order to remove the wireless AP provided using the mobile AP or the illegal MAC address in the positioning process, the Wi-Fi location server may extract only a predetermined number of wireless AP information in order of RSSI.

Thereafter, the Wi-Fi location server may extract (or analyze) the location information of the mobile terminal 500 using at least one wireless AP information extracted from the database. And compares the received information with the received wireless AP information to extract (or analyze) the location information of the mobile terminal 500.

As a method for extracting (or analyzing) the position information of the mobile terminal 500, a Cell-ID method, a fingerprint method, a triangulation method, and a landmark method can be utilized.

The Cell-ID method is a method of determining the position of the mobile station with the strongest signal strength among neighboring wireless AP information collected by the mobile terminal. Although the implementation is simple, it does not cost extra and it can acquire location information quickly, but there is a disadvantage that positioning accuracy is lowered when the installation density of the wireless AP is low.

The fingerprint method collects signal strength information by selecting a reference position in a service area, and estimates the position based on the signal strength information transmitted from the mobile terminal based on the collected information. In order to use the fingerprint method, it is necessary to previously convert the propagation characteristics into a database.

The triangulation method is a method of calculating the position of the mobile terminal based on the coordinates of at least three wireless APs and the distance between the mobile terminals. (Time of Arrival, ToA), Time Difference of Arrival (TDoA) in which a signal is transmitted, and the time difference between the wireless AP and the wireless AP, in order to measure the distance between the mobile terminal and the wireless AP. , An angle at which a signal is transmitted (Angle of Arrival, AoA), or the like.

The landmark method is a method of measuring the position of a mobile terminal using a landmark transmitter that knows the location.

Various algorithms can be utilized as a method for extracting (or analyzing) the location information of the mobile terminal.

The location information of the extracted mobile terminal 500 is transmitted to the mobile terminal 500 via the Wi-Fi location server, so that the mobile terminal 500 can acquire the location information.

The mobile terminal 500 may be connected to at least one wireless AP to obtain location information. At this time, the number of wireless APs required to acquire the location information of the mobile terminal 500 may be variously changed according to the wireless communication environment where the mobile terminal 500 is located.

Next, an example in which a WFD session between a WFD source and a WFD sink is established and a screen sharing service is provided will be described with reference to FIG.

6 is a ladder diagram illustrating a process of providing a screen sharing service by establishing a WFD session between a WFD source and a WFD sink according to an embodiment of the present invention.

One or more of the steps of FIG. 6 may be supported by a Wi-Fi Direct Standard document.

Among the WFDS, the display service may refer to services and applications that enable screen sharing among P2P devices. A P2P service using a display service can be referred to as a WFD device, and a device supporting streaming of multimedia contents through a device P2P link among WFD devices can be referred to as a WFD source, a WFD source device The apparatus for receiving and rendering the multimedia content may be referred to as a WFD sink.

The WFD source 10 may be the mobile terminal 500 shown in FIG. Accordingly, the WFD source 10 may include all components of the mobile terminal 500 shown in FIG.

The WFD sink 20 may be the display device 100 shown in Fig. Accordingly, the WFD sink 20 may include all the components of the mobile terminal 500 shown in FIG.

Referring to FIG. 6, the WFD source 10 may display an image on the screen (S600).

In one embodiment, the image displayed on the screen may be an image of the media content.

5, the image displayed on the screen may include a home screen of the mobile terminal 500, an execution screen of an application installed in the mobile terminal 500, The mobile terminal 500 may be an image being displayed.

Thereafter, the WFD source 10 and the WFD sink 20 may first search for the presence of each other prior to WFD connection setup via WFD DeviceDiscovery (S601). Specifically, the WFD devices can recognize the presence of each other through the probe request frame and the probe response frame including the WFD information element (WFD IE). The WFD information element may include basic information for establishing an optimal connection between WFD devices such as device type and device status. When the WFD device receives a probe request frame containing a WFD IE, it may send a probe response frame containing its WFD IE in response. If the WFD device is associated with an AP and acts as a Wi-Fi P2P device, two or more Wi-Fi transceivers logically operate on one physical device. At this time, WiFi direct transceiver is used for searching WFD device. The probe request frame for the search of the WFD device may include a P2P information element (IE) as well as a WFD IE, which may be decoded by a WiFi direct transceiver.

Then, the WFD source 10 and the WFD sink 20 can search each other's service capabilities prior to the WFD connection setup (S603). Specifically, when one of the WFD devices transmits a service search request frame including the WFD capability as an information sub-element, the other WFD device transmits, in response thereto, a service in which its WFD capability is included as an information sub- A search response frame can be transmitted. The service discovery procedure is an optional procedure, and the WFD device supporting the service discovery procedure can perform the discovery WFD device and service discovery procedure supporting the service discovery procedure. For performing the service search procedure, the probe request frame and the response frame used in the device search procedure may include information indicating whether the WFD device has the capability to support the service search procedure.

The WFD source 10 or the WFD sink 20 may then select the peer WFD device for the WFD connection setup. The peer WFD device to be set up for WFD connection setup may be selected by the user's input or the peer WFD device to be automatically set up for WFD connection setup according to the policy may be selected.

The WFD device may then select the WFD connection setup method with the selected peer WFD device (S605). Specifically, the WFD device can establish a WFD connection with a connection scheme of either Wi-Fi P2P and TDLS (Tunneled Direct Link Setup). Connectivity information and an associated BSSID sub-element that is communicated with the WFD information element.

If a successful WFD setup is performed using Wi-Fi P2P or TDLS between WFD devices, the WFD device may proceed with WFD capability negotiation (S607). Specifically, the WFD source 10 and the WFD sink 20 exchange a message using a Real-Time Streaming Protocol (RTSP) protocol to set a parameter defining the audio / video payload during the WFD session You can decide. The WFD source 10 and the WFD sink 20 can exchange their specification information with each other. The WFD source 10 may query the capabilities of the WFD sink 20 using the RTSP protocol and the WFD sink 20 may send its capability information to the WFD source 10 in response. For example, the WFD sink 20 may send resolution information to the WFD source 10 including resolutions that it can support. For example, the WFD sink 20 may transmit resolution information to the WFD source 10, including resolutions that the resolutions that can be supported by them are 16: 9 or 4: 3 in aspect ratio. Here, the resolution corresponding to 16: 9 or 4: 3 is merely an example, and the resolution may correspond to an aspect ratio longer than the vertical length.

Conversely, the WFD source 10 may also send resolution information, including resolutions that it can support, to the WFD sink 20.

7 to 9 are views for explaining resolution information transmitted from a WFD sink to a WFD source according to an embodiment of the present invention.

The WFD sink 20 may transmit one or more of the resolution tables shown in Figures 7 through 9 to the WFD source 10. [ That is, the WFD sink 20 may transmit to the WFD source 10 a resolution corresponding to an aspect ratio of 16: 9 or 4: 3, and a resolution corresponding to an aspect ratio longer than the vertical length.

The resolution table shown in FIG. 7 may be a table indicating a set of resolutions supported in a handheld device supported by the WFD device and a refresh rate corresponding to the resolution.

The resolution table shown in FIG. 8 may be a table indicating a set of Consumer Electronics Association (CEA) resolutions supported by the WFD device and a refresh rate corresponding to the resolution.

The resolution table shown in FIG. 9 may be a table indicating a set of supported Video Electronics Standards Association (VESA) resolutions supported by the WFD device and a refresh rate corresponding to the resolution.

The WFD source 10 may send a request message M3 querying the capabilities of the WFD sink 20 using the RTSP protocol after performing the connection setup with the WFD sink 20. 7D, the WFD source 10 may send a request message containing the wfd-video-formats parameter 730 to the WFD sink 20. The video format parameter may be a parameter for which the WFD sync 20 queries the supported video resolution. In response to the request message received from the WFD source 10, the WFD sink 20 may send a response message M3 containing the resolution information shown in Figs. 7 to 9 to the WFD source 10. The resolution information may include the resolution tables shown in FIGS. This can be confirmed through the value of the video format parameter 730 included in the response message.

6 will be described again.

The WFD source 10 may transmit the necessary request message to the WFD sink 20 using the RTSP protocol in the WFD capability negotiation process with the WFD sink 20 in operation S609. For example, the WFD source 10 may send a message containing a message to deliver video of video content corresponding to a particular resolution based on the resolution information received from the WFD sink 20 to the WFD sink 20 ). More specifically, the WFD source 10 may select any of a plurality of resolutions that the WFD sync 20 can support, and send a message to the WFD sink 20 to transmit video of the video content with the selected resolution . That is, the WFD source 10 can select any one of the plurality of resolutions included in the resolution information shown in FIGS. 7 to 9, and transmits a message indicating that the selected resolution image is to be transmitted to the WFD sink 20 Lt; / RTI >

The WFD sink 20 may transmit the response message via the RTSP protocol based on the request message received from the WFD source 10 (S611).

The resolution selected by the WFD source 10 in one embodiment may not be included in the resolutions that the WFD source 10 can support. That is, the resolution of the image being displayed by the WFD source 10 may not be included in resolutions that the WFD sink 20 can support. Steps S609 and S611 may be included in the WFD capability negotiation step (S607).

A WFD session (or a mirrorcast session) is established between the WFD source 10 and the WFD sink 20 and the audio and video The content may be streamed (S613).

In one embodiment, the WFD sink 20 may display a full image of the streamed video content.

In the following drawings, it is assumed that the WFD source 10 may be the mobile terminal 500 described in FIG. 5, and the WFD sink 20 is the display device 100 described with reference to FIG.

Accordingly, the WFD source 10 may include all the components of the mobile terminal 500 shown in FIG. Likewise, the WFD sink 20 may also include all the components of the display 100 shown in FIG.

While the audio and video contents are being streamed, the WFD sink 20 receives an enlargement request signal for enlarging some images included in the entire image of the video content (S615).

In one embodiment, the magnification request signal may be a signal for magnifying some of the images included in the entire image of the video content being displayed through the display 180 of the WFD sync 20.

That is, the enlargement request signal may be a signal for enlarging only a specific region around a certain point of the entire image of the video content. Here, one point may be a point where the pointer moved by the remote control device 200 is located.

In one embodiment, the WFD sink 20 may receive an enlargement request signal from the remote control device 200. The remote control device 200 can transmit an enlargement request signal to the WFD sink 20 when a button provided therein is selected.

The function of the WFD sync 20 for enlarging some images of the entire image may be called a magic zoom function, but this is only an exemplary name.

In one embodiment, when the magic zoom function button included in the remote control device 200 is selected, the display device 100 may receive an enlargement request signal from the remote control device 200.

In response to the received enlargement request signal, the WFD sink 20 transmits an enlargement mode request message to the WFD source 10 (S617).

In one embodiment, the magnify mode request message may be a message indicating that the WFD sync 20 is performing a magic zoom function.

The enlargement mode request message may be a message for further transmitting an enlarged image of a part of the image corresponding to the position of the pointer. Specifically, the enlargement mode request message may be a message requesting transmission of an enlarged version of a partial image corresponding to a point where the pointer is located at a point of time when the display apparatus 100 receives the enlargement request signal from the remote control apparatus 200 .

The enlargement mode request message may be a message requesting the WFD sink 20 to further transmit an enlarged image of the partial image corresponding to the position of the pointer as the magic zoom function is performed.

That is, the enlargement mode request message may be a message requesting the transmission of the entire image of the video content and the enlarged image of the partial image corresponding to the position of the pointer within the entire image.

The enlargement mode request message may include position information that identifies an enlarged area of the entire image. The position information of the enlarged area may include the coordinates of the pointer and the coordinates of the enlarged area centered on the pointer. . The position information of the enlarged area will be described later.

In one embodiment, the WFD sink 20 may send a widening mode request message to the WFD source 10 via a User Input Back Channel (UIBC).

The magnify mode request message may include a flag indicating that the WFD sync 20 is performing a magic zoom function.

The WFD source 10 acquires the enlarged area within the entire image of the video content being displayed based on the positional information corresponding to the enlarged area included in the enlargement mode request message received from the WFD sink 20 (S619).

In one embodiment, the WFD source 10 can acquire an enlarged area to be enlarged in the entire image, based on at least one of the coordinates of the pointer and the coordinates of the enlarged area included in the position information corresponding to the enlarged area.

The WFD source 10 transmits the data of the entire image of the video content and the data of the enlarged partial image corresponding to the acquired enlarged area to the WFD sink 20 (S621, S622).

That is, the WFD source 10 can transmit the data of the entire image of the video content and the data of the enlarged partial image corresponding to the enlarged area to the WFD sink 20 at the same time.

The WFD source 10 may transmit the data of the entire image of the video content and the data of the enlarged partial image to the WFD sink 20 through different channels.

In one embodiment, the WFD source 10 may transmit the data of the entire image of the video content and the data of the enlarged partial image to the WFD sink 20 using a Real-Time Streaming Protocol (RTSP) protocol.

The WFD sink 20 displays the entire image based on the received whole image data on the sub-screen of the display unit, and displays the enlarged partial image based on the data of the enlarged partial image on the sub-screen of the display unit (S623).

In one embodiment, the main screen and the sub screen of the display unit may be different depending on the size.

For example, the size of the main screen of the display unit of the WFD sync 20 may be larger than the size of the sub screen.

The WFD sink 20 may move the entire video content to a location where the letterbox is displayed. At the same time, the WFD sink 20 can display the enlarged partial image on the remaining screen of the display unit while moving the entire image of the video content to the position where the letter box is displayed.

10 is a flowchart for explaining a UIBC connection between a sink device and a source device.

Referring to FIG. 10, a source device and a sink device for sharing contents can be connected (Wi-Fi Direct Connection Setup) through a Wi-Fi direct network proposed by Wi-Fi Alliance. In this case, And the remaining electronic devices can respond to the role of a sink electronic device that receives and outputs the content.

Meanwhile, the source device and the sink device can exchange capability information and perform capability exchange and negotiation.

For example, when content sharing is performed according to the Wi-Fi CERTIFIED Miracast (TM) standard introduced by the Wi-Fi Alliance, the first time a Miracast connection is made, Capability Negotiation A video format and an audio format to be transmitted are determined.

Meanwhile, Miracast uses Option of User Input Back Channel (UIBC) to control the source device using the user input of the sink device during Miracast connection. .

For example, when a smartphone is a source device and a TV is connected to a sink device, not only a sink device but also a source device can be controlled by a control means such as a mouse, a keyboard, and a remote control device connected to the TV.

Capability for UIBC can also be determined at the time of function negotiation. That is, after confirming the UIBC category such as a mouse and a keyboard supported by the sink device, the source device determines a UIBC category that is commonly supported by the sink device and the source device, and services the same.

Then, the sink device transmits UIBC information according to the user input to the source device, and the source device can perform the corresponding operation.

11 is a flowchart showing an embodiment of a method of operating a source device according to an embodiment of the present invention.

Referring to FIG. 11, a source device according to an embodiment of the present invention may be connected to a sink device and a wired / wireless network. The source device according to an exemplary embodiment of the present invention may be connected to an external electronic device through a wired or wireless network for content sharing mode operation, which may transmit content to another device or receive content to another device (S 1110).

A source device according to an embodiment of the present invention can request or respond to a network connection to another device based on a command input of the user, for example, a command for executing the corresponding menu or application, You can enter.

Here, the content sharing mode may mean a mode in which a source device can transmit content to an external device connected to a wireless network or receive content transmitted from an external device.

In addition, the content sharing mode can use the content reproduced by any one of the plurality of devices connected to the mirroring mode in which the connected devices display the same screen, from other devices, and perform other operations in the content sharing state And a dual screen dual play (DSDP) mode.

In addition, the source device according to the present invention can be applied to a public multimedia transmission technology such as Wi-Fi CERTIFIED Miracast (TM), WiFi Display (WFD) and Wireless Display (WiDi) And can share content according to that specification.

Meanwhile, the data for content sharing may be directly transmitted to external devices via wired or wireless communication according to various communication standards, or may be transmitted to an external device through a separately connected web server.

Meanwhile, the source device according to an embodiment of the present invention can be wirelessly connected to a sink device via a Wi-Fi Direct Network. A Wi-Fi direct network is a network in which two devices can be peer-to-peer connected via Wi-Fi, and devices connected to a Wi-Fi direct network can acquire resource, capability information, and the like.

Meanwhile, the source device according to the embodiment of the present invention can perform capability negotiation with the sink device (S1120)

In the function negotiation step S1120, the source device and the sink device according to an embodiment of the present invention can determine a video format and an audio format of contents to be shared and transmitted.

The source device according to an embodiment of the present invention determines a video format and an audio format to be transmitted through the function negotiation S1120 in operation S1120 and outputs the content content to the sink device The content data including video data, audio data, and the like can be transmitted to the sink device in a format based on the function negotiation (S1130)

A source device according to an exemplary embodiment of the present invention may transmit control image data including one or more control menu buttons to a sink device (S1140). That is, the source device may transmit control image data , A control event may be generated, or data may be transmitted to the sink device in advance in the content sharing mode.

Here, the control image data may include the name of the control menu button and the position information in the control image of the control menu button.

The source device can transmit the control image together with the position information of each menu button. For example, the control names 'Resolution' and 'Media' of each menu button and the position information (xy width hegith) in the control image can be transmitted as follows.

Wfd_uibc_control_image: Resolution 10 10 30 20, [0168] Media 10 40 30 20

Further, depending on the embodiment, the control image data may further include size information of the control image. That is, in addition to the relative position coordinates of each menu button in the control image, further use of the full size information of the control image can help to grasp the position that the user inputs more quickly.

Also, according to an embodiment, the control image data may include position information of the control image, the name of the control menu button, and the position information of the control menu button. The source device can transmit the control image together with the position information of the control image, the name of each menu button, and the position information. In this case, the position input by the user to the sink device can be compared with the position information of the control image and the button position information in the control image in absolute coordinates to discriminate the control command selected by the user.

On the other hand, the sink device receiving the control image data can receive a user's input by providing a control UI screen based on the received data. The user may select a control menu button contained within the UI screen to select an instruction to control the source device according to an embodiment of the present invention. Therefore, it is possible to input various control commands without limitation depending on the configuration of the control menu button.

For example, the control image data may include a control menu button corresponding to a control command associated with the content sharing mode, i. E., Configure the control menu button with control commands frequently used in the content sharing mode.

 The control image data may include a control menu button corresponding to a control command determined according to the type of the sink device or the type of content shared with the sink device. That is, the control menu button can be configured with optimized control commands according to the type of content being used in the content sharing mode.

On the other hand, the type of content can be discriminated based on the format of received data, received from the partner device, extracted from the metadata, or determined through various known screen analysis algorithms.

On the other hand, depending on the embodiment, the control image data may be transmitted together with the data to be transmitted at the function negotiation (S1120).

Thereafter, the source device can receive the user input information for the control image data from the sink device (S1150), and can perform the control function corresponding to the user input information (S1160)

Herein, the user input information may include a user's input position coordinate with respect to the control image data performed in the sink device.

When the user selects the control menu button in the corresponding image in the sink device, the coordinates selected by the source device are transmitted. When the source device receives the data, the selected menu button is discriminated and the corresponding operation can be performed.

Alternatively, the sink device may determine the selected menu button according to the coordinates of the input position of the user with respect to the control image data directly performed, and may transmit the determined menu button information to the source device as user input information.

In this case, the source device can perform an operation corresponding to the received menu button information.

On the other hand, the user input information can be received as UIBC information through a user input back channel (User Input Back Channel).

According to the embodiment of the present invention, the UIBC for controlling the source device using the user interface of the sink device can be expanded without limitation even in a state of mirroring the source device screen to the sink device with the mirror cast.

The source device according to an embodiment of the present invention can deliver the control UI image and related data to the sink device.

The source device can receive a selection position on the control UI image from which the user has selected an item on the control UI image from the sink device and perform control accordingly.

Therefore, according to the present invention, there is a technical effect that control of the source device using the control UI image output from the sink device enables more scalable and convenient control of the source device.

12 and 13 are diagrams for explaining a method of controlling a source device and a sink device according to an embodiment of the present invention.

Referring to FIG. 12, the sink device and the source device may be connected to a wired / wireless network for content sharing. For example, a sink device and a source device can be wirelessly connected via a Wi-Fi Direct Network.

The sink device and the source device can determine features for content sharing and determine functions through capability negotiation.

Meanwhile, FIG. 13 illustrates that the screen of the mobile terminal 1100 is mirrored by the display device 1200. FIG. If the user wants to change the resolution setting, the mobile terminal 1100, which is a source device, must be directly manipulated, and it must be set in the order of control panel> display> screen resolution (1310).

However, according to an embodiment of the present invention, a control image 1300 including a resolution menu button 1310 may be transmitted to the display device 1200, as shown in FIG.

Further, the source device can transmit the control image together with the position information of each menu button.

For example, if the position information (xy width height) in the control image of the control name 'Resolution' button 1310, the 'Media' button 1320 and the control menu button of each menu button is set to 'Wfd_uibc_control_image: Resolution 10 10 30 20, and Media 10 40 30 20 '. Further, width, height information, size information, and the like of the control image can be further transmitted.

Referring to FIG. 13, the control image 1300 may have positional information of C2 (50,0), C3 (0,100), and C4 (50,100) with the left top C1 (0,0) as a coordinate reference. The 'Resolution' button 1310 also displays a1 (10,10), a2 (40,10), a3 (10,30), a4 (40,30) And the 'Media' button 1320 can also be drawn in the control image 1300 in the same manner as the coordinates of the upper left corner C1 (0,0).

On the other hand, the display device 1200 receives, from the mobile terminal 1100, content data including video data, audio data and the like in a format based on function negotiation and control image data including one or more control menu buttons .

On the other hand, the display device 1200 may display a control image 1300 including one or more control menu buttons as well as a mirroring screen.

The user can select a menu button for controlling the mobile terminal 1100 by using the remote control device 400 of the display device 1200. [

14 is a flowchart for explaining another example of a method for controlling data transmitted from a source device to a sink device according to an embodiment of the present invention.

In an embodiment, it is assumed that the sink device supports the capability of transmitting information (e.g., message type) about the user input received via the predetermined user input device to the source device via the UIBC. User Input Back Channel (UIBC) means a channel through which a sink device transmits information on a user input received from a user (for example, a user input device) to a source device based on Wi-Fi Direct.

The controller 180 of the mobile terminal 1100 as the source device requests the display device 1200, which is a sink device, to inquire information on whether or not the performance of the UIBC is supported (S1410).

In response to the request, the control unit 270 of the display device 1200 transmits information on whether or not user input is supported through the UIBC to the mobile terminal 1100 through the network interface unit 230 (S1420) . In accordance with the embodiment, the controller 270 may transmit to the mobile terminal 1100, in response to the request, parameter information necessary for processing the user input, together with information on whether the user input is supported through the UIBC .

The control unit 270 of the display device 1200 transmits data (for example, streaming data) received from the mobile terminal 1100 through the network interface unit 230 to the display unit 280 And / or the audio output unit 285 via the audio output unit 285.

Meanwhile, the storage unit 240 may store user control input data that can be transmitted to the mobile terminal 1100 through the UIBC when mirroring. For example, the user control input of the user input device may include information such as movement of a mouse or a touch pad, information such as a click, a touch, a zoom magnification, a scroll amount, a rotation amount, .

The control unit 270 of the display device 1200 receives the user control input signal received through the user input interface unit 250 and converts the received user control input signal into data for transmission to the mobile terminal 1100 (S1430).

The control unit 270 may control the network interface unit 230 to transmit the message including the command of the user control input corresponding to the converted data to the mobile terminal 1100 through the UIBC.

The control unit 180 of the mobile terminal 1100 receives a message from the display device 1200 through the wireless communication unit 110 and transmits the message to the mobile terminal 1100 in response to a user control input corresponding to a command included in the message. The function can be controlled (S1450).

The control unit 180 processes the received message based on the parameters necessary for processing the user control input received from the display device 1200 and can execute a predetermined function corresponding to the command.

The control unit 180 of the mobile terminal 1100 may control the wireless communication unit 110 to transmit data reflecting the control result to the display device 1200 (S1460). For example, the control unit 180 executes a function corresponding to an instruction word, outputs a screen including a result of executing the function to the display unit 151, and outputs data corresponding to the output screen to the display device 1200 ). In this way, the screen of the display unit 151 including the control result can be mirrored on the display unit 280 of the display device 1200.

FIG. 15 shows an example in which a screen displayed on the mobile terminal is displayed on the TV using the DLNA function.

As shown in FIG. 15, when the screen displayed on the mobile terminal is displayed on the TV screen by the DLNA function, a browsing toolbar 1550 suitable for the TV screen is provided. The browsing toolbar 1550 is displayed only on the TV screen, not on the mobile terminal screen, such as the previous / next history movement and the address window.

When a browser screen is displayed on the TV screen, a mouse cursor 1551 is displayed. The mouse cover disappears automatically when there is no input for 5 seconds and reappears when the screen of the mobile terminal is touched.

16 and 17 show an example of mode switching of a mobile terminal when a mobile terminal and a TV are connected by DLNA in the present invention.

When the DLNA function is applied between the mobile terminal and the TV, the controller 180 switches the mode of the mobile terminal to the mouse pad mode as shown in FIG. 16 so that the user can use the mobile terminal as an input device. The mouse cursor 1551 is moved by a user input on the mouse pad, that is, a touch input. When the mouse cursor 1551 is located at a specific position on the browser screen, the user can select specific content by touch input. In addition, the switching to the mouse pad mode may be automatically performed when the DLNA function between the mobile terminal and the TV is set.

When the user moves the mouse cursor 1551 to the text field using the mouse pad, the controller 180 switches the mode of the mobile terminal from the mouse pad mode to the 'keypad input mode' And automatically displays the keypad on the screen of the mobile terminal. The fact that the user moves the mouse cursor 1551 to the text field means that the user inputs text, so the user can display the keypad to perform text input. At this time, an 'X' icon is displayed on the top of the keypad to select whether or not to execute the keypad input mode. Accordingly, when the user selects the 'X' icon, the mouse pad mode returns to the previous mode. The overall operation of the DLNA web browsing through the mobile terminal and the TV is shown in FIG.

19 is a flowchart illustrating an operation method of a terminal according to an embodiment of the present invention.

The control unit 170 of the mobile terminal 100 outputs the first window indicating the execution screen of the first program to the entire screen of the display unit 180 (S1901). In one embodiment, the program may correspond to any one of an application program and an application installed in the terminal 100.

The control unit 170 of the terminal 100 receives a screen division request for dividing the entire screen into the first area and the second area (S1903), and in response to the received screen division request, And outputs the second window to the second area (S1905).

In one embodiment, the screen division request may be a request to divide the entire screen of the display unit 180 into a first area and a second area having the same area, but the present invention is not limited thereto. That is, the area of the first area and the area of the second area may be different from each other. The second window may be a multitasking screen for providing a multitasking environment to the user.

In one embodiment, the second window may include a program list indicating programs other than the first program and an external input list indicating an external input connected to the terminal 100 in a wired or wireless manner.

In another embodiment, a screen of a peripheral device connected to the terminal 100 via a wired or wireless connection may be output on the second window. This will be described later.

20 is a diagram for explaining an embodiment of outputting an execution screen of a first program as a whole screen according to an embodiment of the present invention.

Referring to FIG. 20, the controller 170 may control the display unit 180 to display the first window 400 corresponding to the execution screen of the first program on the full screen 300. [0064] FIG. For example, the first window 400 may be provided with a web site connection screen.

21 is a view for explaining an embodiment in which a second window is additionally output by dividing the entire screen into a first area and a second area according to an embodiment of the present invention.

20, when the screen division request is received, the controller 170 can divide the entire screen 300 into the first area 310 and the second area 330 as shown in FIG. 21 have.

In one embodiment, when the terminal 100 is a PC or a notebook, a screen division request may be generated by an input of simultaneously pressing a control key and a tab key provided on the keyboard. Here, the screen division request is generated by inputting a control key and a tab key at the same time, which is an example only, and can be changed according to the setting of the user.

In another embodiment, when the terminal 100 is a TV such as a smart TV, a screen division request may be generated by an input of pressing a specific button provided on the remote control device 200. [

The controller 170 displays a first window 400 indicating the execution window of the first program in the first area 310 and a second window window 500 for providing a multitasking environment in the second area 320 The display unit 180 may be controlled. The size of the first window 400 may be reduced according to the size of the first area 310.

Here, the area of the first area 310 and the area of the second area 330 are described as being the same, but this is merely an example, and the areas may be different from each other.

On the second window 500, a program list 510 and an external input list 530 can be displayed. The program list 510 may include one or more program items 511-516.

In one embodiment, each program item may represent a recent execution picture corresponding to a program recently executed in the terminal 100 or a thumbnail image corresponding to a recently executed program. The one or more program items 511 to 516 may be arranged in order according to the latest execution time. That is, the first program item 511 is the item corresponding to the most recently executed program, and the second program item 512 to the sixth program item 516 may follow.

In another embodiment, each program item may represent a thumbnail image corresponding to a program registered as a favorite or a favorite execution screen corresponding to programs registered as favorites.

The external input list 530 may include one or more external input icons. Each external input icon may represent an external input wirelessly or wired to the terminal 100. For example, the first external input icon 531 may be an icon representing an external input connected through a screen mirroring technique. The screen mirroring technology is a technology for sharing data between terminals by transmitting data through a short distance wireless communication technology such as Wi-Fi and Bluetooth. The second external input icon 533 may be an icon representing an external input connected through the HDMI (High Definition Multimedia Interface) standard. The third external input icon 535 may be an icon representing a wired or wireless connected TV. Although not shown in FIG. 21, the external input list 530 may further include peripheral devices such as a DVR connected to the terminal 100, a Blu-ray player, a USB memory, a terminal for transmitting media content using a Widi (wireless Display) And may further include external input icons.

On each external input icon, one or more of a name identifying the external input, a thumbnail image of the media content recently played through the external input, and a name of an externally input peripheral device may be displayed.

The external input icons 531 to 535 may be arranged in order according to the point in time when the terminal 100 is recently connected.

For example, if the first external input icon 531 is the icon of the external input most recently connected to the terminal 100, and the second external input icon 533 and the third external input icon 535 can follow have.

In yet another embodiment of the present invention, the external input list 530 may display thumbnail images of media content reproduced through an external input instead of an external input icon.

In yet another embodiment of the present invention, thumbnail images of the media content reproduced through the external input may be displayed on each external input icon.

The control unit 170 of the terminal 100 transmits a selection request for selecting one of one or more program items representing a program contained on the output second window and one or more external input icons representing an external input (S1907), and outputs one or more windows of a first window, which is an execution window corresponding to the selected program item, and a second window, which is an external input window, corresponding to the selected external input icon according to the received selection request (S1909).

22 to 23 show examples of screens provided when one or more program items are selected in the program list on the second window according to an embodiment of the present invention.

Especially. FIG. 22 shows an embodiment of a screen provided when one program item is selected in the program list, and FIG. 23 shows an embodiment provided when two program items are selected in the program list.

When a request to select the first program item 511 in the program list 510 of FIG. 21 is received, the control unit 170 controls the third window 600 (FIG. 5) to be the execution window of the program corresponding to the first program item 511 May be displayed on the second area 330. [0051] In this case, the second window 500 that has been displayed in the second area 330 may not be displayed due to the third window 600. In one embodiment, when the terminal 100 is a PC or a notebook, a request to select the first program item 511 may be made by pressing a mouse click or a control key on the keyboard and a Tab Key Lt; / RTI > In another embodiment, when the terminal 100 is a TV, a request to select the first program item 511 may be performed via the remote control device 200. [

The user can easily execute another program through the information of the programs displayed on the second window 500 to perform multitasking work.

When a request to select the first program item 511 and the second program item 513 is received from the program list 510 of FIG. 21, the control unit 170 determines whether the program corresponding to the first program item 511 It is possible to control the display unit 180 to display the fourth window 700 as the execution window of the program corresponding to the third window 600 and the second program item 513 in the second area 330 have. The second region 330 may be divided into a first sub-region 331 and a second sub-region 333 when a request to select a first program item 511 and a second program item 513 is received have. The third window 600 may be displayed on the first sub region 331 and the fourth window 700 may be displayed on the second sub region 700. [ In this case, the second window 500 that has been displayed in the second area 330 may not be displayed due to the third window 600 and the fourth window 700.

In one embodiment, when the terminal 100 is a PC or a notebook, a request to select a first program item 511 and a second program item 513 may be made by pressing a control key on the keyboard, May be generated by an input selecting the first program item 511 and the second program item 513. In another embodiment, when the terminal 100 is a TV, a request to select the first program item 511 and the second program item 513 may be performed via the remote control device 200. [

The user can easily execute another program through the information of the programs displayed on the second window 500 to perform multitasking work.

According to another embodiment of the present invention, there is provided a program for providing a multitasking environment while displaying execution screens of a program

A second window can be output again. In addition, an execution screen of another program can be additionally output through the output second window.

24 to 26 are views for explaining an embodiment for additionally outputting an execution screen of another program through a second window for providing a multitasking environment according to an embodiment of the present invention.

Fig. 24 (a) is the same as Fig. 22. 24 (a), when the second window output request for outputting the second window 500 for providing the multitasking environment is received, the controller 180, as shown in FIG. 24 (b) The display unit 180 may be controlled to additionally display the second window 500 on the full screen 300. [ Here, it is assumed that the full screen 300 includes a fourth area 351, a fifth area 353, and a sixth area 355. The control unit 180 displays the first window 400 as the execution window of the program corresponding to the first program in the fourth area 351 of the full screen 300 according to the second window output request, The display unit 180 displays the third window 600 as the execution window of the program corresponding to the first window 351 in the fifth area 353 and displays the second window 500 in the sixth area 355 Can be controlled.

If the third program item is selected on the second window 500, the control unit 180 can control the display unit 180 to display the execution window of the program corresponding to the third program item in the sixth area 355 have.

In one embodiment, when the terminal 100 is either a PC or a notebook, a request to output the second window 500 in FIG. 24 may include a control key and a tab key provided on the keyboard Can be generated by an input that is pressed simultaneously.

In another embodiment, when the terminal 100 is a TV, such as a smart TV, a request to output the second window 500 may be generated by an input of pressing a specific button provided on the remote control device 200. [

Fig. 25 (a) is the same view as Fig. 24 (b).

25 (a), when a layout change request for changing the layout of the screens constituting the entire screen 300 is received, the control unit 180 controls the first window 400, the second window 500, and the third window 600 can be changed. That is, in response to the layout change request, the controller 180 sets the first window 400 to the first area 310, the third window 600 to the first sub-area 331 of the second area 330, It is possible to control the display unit 180 to display the second window 500 in the second sub-area 333 of the second area 300. In one embodiment, if the terminal 100 is a PC or a notebook, a layout change request may be generated by an input that simultaneously presses the control key and direction key of the keyboard. In another embodiment, when the terminal 100 is a TV, a layout change request may be received via the remote control device 200. [

If the third program item 513 is selected on the second window 500 of FIG. 25 (b), the controller 180 determines that the third program item 513 is selected in the second sub-area 330 of the second area 300, It is possible to control the display unit 180 to display the fourth window 700, which is an execution window of the program corresponding to the second window 513.

Next, with reference to FIG. 27 to FIG. 28, a process of changing the layout of the entire screen 300 according to another embodiment of the present invention and performing a multitasking operation through the changed layout will be described.

27 to 28 are views for explaining an embodiment in which the layout constituting the entire screen 300 is changed according to another embodiment of the present invention, and multitasking work can be performed through the changed layout.

Fig. 27 (a) is the same view as Fig. 27A, when the layout expansion request for expanding the layout is received, the controller 180 enlarges the size of the second area 330 as shown in FIG. 27B, The size of the first area 310 can be reduced. Accordingly, the size of the first window 400 can be reduced by the size of the reduced first region 310, and the size of the second window 500 can be enlarged by the size of the expanded second region 330 . In one embodiment, when the terminal 100 is a PC or a notebook, a layout enlargement request may be generated by an input that simultaneously presses a control key, a shift key, and a left directional key of the keyboard. In another embodiment, when the terminal 100 is a TV, a layout enlargement request can be received via the remote control device 200. [

On the other hand, when an input for simultaneously pressing the control key, the shift key, and the right direction key of the keyboard is received, the control unit 180 can reduce the length of the second window 500. The control unit 180 may increase the vertical length of the second window 500 when an input that simultaneously presses the control key, shift key and up key of the keyboard is received. The control unit 180 can reduce the vertical length of the second window 500 when an input for simultaneously pressing the control key, the shift key, and the downward key of the keyboard is received.

Fig. 28 (a) is the same view as Fig. 27 (b). 28A, when a request to select the first program item 511 included in the program list 510 is received, the controller 180 executes the program corresponding to the selected first program item 511 It is possible to control the display unit 180 to display the third window 600, which is a screen, in the enlarged second area 330. [

Next, an example of a screen provided when an external input icon for screen mirroring is selected from the external input list on the second window will be described.

29 to 30 show an example of a screen provided when an external input icon for screen mirroring is selected from an external input list on the second window according to an embodiment of the present invention.

When a request to select the first external input icon 531 for screen mirroring is received from the external input list 530 output to the second window 500 in Figure 29A, It is possible to control the display unit 180 to display the screen mirroring list screen 800 in the second area 330 of the full screen 300 as shown in (b) of FIG.

In one embodiment, the scroll mirroring list screen 800 may include a mobile terminal list 810 that is currently connected to the terminal 100 and can share a screen. The mobile terminal list 810 may include at least one of a name of each mobile terminal and a user name of each mobile terminal.

In another embodiment, the scroll mirror list screen 800 may include a mobile terminal list 810 including a mobile terminal connected to the terminal 100 and a mobile terminal that has been connected.

The terminal 100 may perform screen mirroring with the selected one or more mobile terminals when one or more mobile terminals included on the mobile terminal list 810 are selected. This will be described later.

29 (a), when the controller 180 receives a request to select the first external input icon 531 for screen mirroring from the external input list 530 output to the second window 500, The display unit 180 may be controlled to display the screen mirroring list screen 800 in the second area 330 of the full screen 300 as shown in FIG.

Referring to FIG. 30, the screen mirroring list screen 800 may include a mobile terminal list 830 that is currently connected to the terminal 100 and can share a screen. The mobile terminal list 830 may include at least one of a screen of each mobile terminal, a name of each mobile terminal, and a user name of each mobile terminal. For example, the mobile terminal list 830 may include a screen 831 of a first mobile terminal, a screen 833 of a second mobile terminal, and a screen 835 of a third mobile terminal. In one embodiment, the screen of each mobile terminal may be the current screen of the mobile terminal connected to the terminal 100. In another embodiment, the screen of each mobile terminal may be a screen of the mobile terminal at the time when the connection with the mobile terminal 100 is terminated. When one or more screens are selected from the mobile terminal list 830, the controller 180 may output the selected one or more mobile terminal screens on the full screen 300. [ Accordingly, the terminal 100 may perform screen mirroring with the mobile terminal corresponding to the screen of the selected mobile terminal.

Next, an example of a screen provided when an external input icon indicating HDMI input is selected from the external input list is shown.

FIGS. 21 to 32 show examples of screens provided when an external input icon representing an HDMI input is selected from an external input list on the second window according to an embodiment of the present invention.

31 (a), when a request to select the second external input icon 533 included in the external input list 530 output to the second window 500 is received, the display unit 180 is controlled to display the current screen 900 of another terminal connected to the terminal 100 through the HDMI standard in the second area 330 of the full screen 300 as shown in FIG. can do. The other terminal may be any one of a PC and a notebook. The screen 900 of the other terminal will now be described with reference to FIG.

31 (b), when a request to output the second window 500 is received on the first area 310, the controller 180 controls the first area 310, A second window 500 for displaying the first window 400 in the first sub region 311 of the first region 310 and providing a multitasking environment to the second sub region 313 of the first region 310, It is possible to control the display unit 180 to display it. A request to output the second window 500 on the first area 310 may be a screen division request to divide the first area 310 into two sub-areas.

In the second window 500, a screen division icon (PBP, Picture By Picture icon, 550) indicating a screen division function may be displayed. When a request to select the screen division icon 550 is received in FIG. 32 (a), the controller 180 terminates the screen division function as shown in FIG. 32 (b) The display unit 180 may be controlled such that only the first window 400 is displayed.

Next, an example of a screen provided when an external input icon representing TV input is selected from the external input list is shown.

FIG. 33 shows an example of a screen provided when an external input icon indicating a TV input among the external input list on the second window is selected according to an embodiment of the present invention.

When a request to select a third external input icon 535 indicating TV input is received from the external input list 530 output to the second window 500 in Figure 33A, The display unit 180 can be controlled to display the TV screen 1000 in the second area 330 of the full screen 300 as shown in (b) of FIG. That is, the controller 180 controls the display unit 180 to display the TV screen 1000 connected to the terminal 100 in the second area 330 according to a request to select the third external input icon 535 . The TV program 1000 may display a program currently being reproduced on the TV.

Hereinafter, it is assumed that the source device is a mobile terminal and the sink device is a display device or a TV.

34 to 36 are diagrams for explaining an example in which content is executed through a DLNA during mirroring between a source device and a sink device according to an embodiment of the present invention.

As shown in FIG. 34 (a), a source device and a sink device according to an embodiment of the present invention can connect mirroring. In addition, the sink device executes and displays its application on the first layer 3410 during the mirroring connection, and the second layer 3420 overlaps with the first layer 3410 in a certain area, and the second layer 3420 The screen of the source device can be displayed.

The user of the sink device may select one of the icons displayed on the second layer 3420 to execute the content. 34B, the sink device may access the content of the source device via the DLNA connection and display the content selected by the user through the application in the first layer 3410, as shown in FIG. 34B.

35A and 35B, the user can control the source device on the second layer 3420 in a state in which the content via DLNA is displayed on the first layer 3410 Do.

36 (a) through 36 (c) as another embodiment, when the user selects one content from the content list displayed on the second layer 3620, the user selects one content on the second layer 3620 The content can be executed. When the user selects the second layer 3620, the sink device executes the corresponding content through the DLNA to display the content on the first layer 3610, and the second layer 3620 displays the content in its original state The content list can be displayed.

38 is a view for explaining an example of executing content through Deep Linking during mirroring between a source device and a sink device according to an embodiment of the present invention.

As shown in FIG. 38, a source device and a sink device according to an embodiment of the present invention can perform a mirroring connection. The sink device may display a screen of the application on the first layer 3810 and display the mirroring screen of the source device on the second layer 3820 in a state of being overlaid with the first layer 3810 . At this time, if the user executes the web content having the web address information in the source device, the corresponding web address information can be transmitted from the source device to the sink device. The sink device can execute the corresponding web content on the first layer 3810 through its own application by applying a deep linking technique using the corresponding web address information. In addition, the user can use the mirroring screen of the source device on the second layer 3820 while the web content is being executed on the first layer 3810.

FIG. 39 is a view for explaining an example of displaying a mirroring icon when mirroring is stopped between a source device and a sink device according to an embodiment of the present invention. FIG.

As shown in FIG. 39, the source device 3910 and the sink device 3920 according to an embodiment of the present invention can perform a mirroring connection. At this time, if the user stops mirroring, the sink device 3920 can display the mirroring icon 3930 on the screen. When the user desires to reconnect the mirroring, the user can simply perform the reconnection by selecting the mirroring icon 3930 displayed on the sink device.

According to an embodiment of the present invention, the above-described method can be implemented as a code readable by a processor on a medium on which a program is recorded. Examples of the medium that can be read by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and may be implemented in the form of a carrier wave (e.g., transmission over the Internet) .

The source device and the sink device or the mobile terminal and the display device described above can be applied not only to the configuration and the method of the embodiments described above but also to all the embodiments of the present invention so that various modifications can be made. Or some of them may be selectively combined.

Claims (15)

In the sink device,
A network interface unit for communicating with the source device in a Wi-Fi direct mode;
A display unit for displaying an image of content received from the source device; And
And a control unit,
Wherein,
Displaying a first application content on a first layer of the display unit,
Displaying a second application content received from the source device on a second layer overlaid in a region of the first layer, and
And control to display the third application content on the first layer based on the content information transmitted from the source device,
Sink device.
The method according to claim 1,
Wherein the first layer is located in the entire area of the display unit,
Sink device.
The method according to claim 1,
Wherein the first application is an application installed in the sink device,
Sink device.
The method according to claim 1,
The content information transmitted from the source device is DLNA (Digital Living Network Alliance) content information,
Sink device.
5. The method of claim 4,
The third application is an application installed in the sink device,
Wherein,
And to output the content corresponding to the DLNA content information through the third application,
Sink device.
The method according to claim 1,
The content information transmitted from the source device is transmitted through a User Input Back Channel (UIBC)
Sink device.
The method according to claim 1,
The network interface unit,
Receiving a coordinate table of the source device when performing a WiFi direct communication with the source device,
Sink device.
The method according to claim 1,
If the third application is an application installed in the source device,
Wherein,
And controlling the mirroring image to be displayed on the first layer and the second layer,
Sink device.
A control method for a sink device,
Performing WiFi direct communication with the source device;
Displaying a first application content on a first layer of the screen;
Displaying a second application content received from the source device on a second layer overlaid on a predetermined area of the first layer; And
And displaying the third application content on the first layer based on the content information transmitted from the source device.
A method of controlling a sink device.
10. The method of claim 9,
The first layer being located in the entire area of the screen,
A method of controlling a sink device.
10. The method of claim 9,
Wherein the first application is an application installed in the sink device,
A method of controlling a sink device.
10. The method of claim 9,
The content information transmitted from the source device is DLNA (Digital Living Network Alliance) content information,
A method of controlling a sink device.
13. The method of claim 12,
The third application is an application installed in the sink device,
The sink device includes:
And outputting the content corresponding to the DLNA content information through the third application,
A method of controlling a sink device.
10. The method of claim 9,
The content information transmitted from the source device is transmitted through a User Input Back Channel (UIBC)
A method of controlling a sink device.
10. The method of claim 9,
Wherein the performing the WiFi direct communication with the source device comprises:
And receiving a coordinate table of the source device.
A method of controlling a sink device.

Images