WO2021118097A1

WO2021118097A1 - User terminal device and method for controlling same

Info

Publication number: WO2021118097A1
Application number: PCT/KR2020/016339
Authority: WO
Inventors: 박희성; 박대희; 송상곤; 이정표
Original assignee: 삼성전자주식회사
Priority date: 2019-12-10
Filing date: 2020-11-19
Publication date: 2021-06-17
Also published as: KR20210073094A

Abstract

Disclosed is a user terminal device. The user terminal device comprises: a communication unit; a display; a sensor; and a processor which, when a motion of the user terminal device caused by a gesture of a user is detected through the sensor, determines the gesture of the user on the basis of the motion of the user terminal device. If the gesture of the user is determined to be a preset gesture, the processor receives information about an object included in an image displayed on the display device from the display device through the communication unit, and displays information about the object and information about a plurality of applications related to the object on the display. If a user command for selecting one of the plurality of applications is received, the processor displays a screen provided by the selected application on the display on the basis of the information about the object.

Description

User terminal device and its control method

The present disclosure relates to a user terminal device and a control method thereof, and more particularly, to a user terminal device operating based on a user's gesture and a control method thereof.

A user who views an image through a display device such as a TV may be interested in an object included in the image while watching the image. For example, a user watching a drama may be interested in actors, cars, buildings, etc. included in the image.

In this case, it is common for the user to execute an application for web search installed in a user terminal device such as a smart phone and search for an object. However, when the user does not know the name or name of the object included in the image, there is a problem in that the user cannot search for the object.

Meanwhile, in recent years, in order to solve this problem, a smart TV has been developed that divides the screen to display a main image such as a drama in a first area of the display and displays an additional image related to the main image in a second area of the display. have. However, this may interfere with the user's viewing of the image due to the reduction in the size of the main image.

The present disclosure has been devised to solve the above problems, and an object of the present disclosure is to provide a user terminal device capable of providing additional information related to an image being reproduced on a display device, and a method for controlling the same.

In a user terminal device according to an embodiment of the present disclosure for achieving the above object, when a movement of the user terminal device by a communication unit, a display, a sensor, and a user's gesture is detected through the sensor, the user terminal device The user's gesture is determined based on the user's gesture, and when it is determined that the user's gesture is a preset gesture, information on an object included in the image displayed on the display device is received from the display device through the communication unit, and the object information on and information on a plurality of applications related to the object are displayed on the display, and when a user command for selecting one of the plurality of applications is received, the selected application based on the information on the object It may include a processor for displaying the provided screen on the display.

In addition, the processor displays information on the object on the display, and when a user command for selecting the object is received, information on the object and information on a plurality of applications related to the object on the display can be displayed

In addition, the processor displays an image in which a UI corresponding to the object is superimposed on the image displayed on the display device, and when a user command for selecting the UI is received, information on the object and the Information on a plurality of applications related to the object may be displayed on the display.

In addition, the processor may obtain information on the number of executions of a plurality of applications based on the user's history information, and display information on the plurality of applications on the display based on the information on the number of executions. have.

Then, the processor receives, from the display device, a plurality of image frames displayed on the display device, and displays an indicator for selecting one image frame among the plurality of image frames on the display, through the indicator , when a user command for selecting one image frame from among the plurality of image frames is received, information on an object included in the selected image frame and information on a plurality of applications related to the object may be displayed on the display. have.

And, when the user's gesture is determined to be a gesture for executing a specific application, the processor determines an object necessary for executing the function of the specific application from among a plurality of objects included in the image displayed on the display device, , a screen provided by the specific application may be displayed on the display based on the object.

In addition, when the specific application is a call application, the processor determines an object representing a phone number from among a plurality of objects included in the image displayed on the display device, and a screen provided by the call application based on the phone number may be displayed on the display.

In addition, when the specific application is a map application, the processor determines an object indicating a location from among a plurality of objects included in the image displayed on the display device, and displays a screen provided by the map application based on the location. can be displayed on the display.

In addition, when the user's gesture is determined as a gesture for screen capture, the processor receives an image displayed on the display device from the display device, and displays a screen provided by a photo application on the display based on the image. can be displayed

In addition, when the user's gesture is determined as a gesture for executing a music application, the processor receives an audio signal output from the display device from the display device, and outputs the audio signal from the display device based on the audio signal The title of the music may be determined, and the music application may be executed based on the title of the music.

On the other hand, the control method of the user terminal device according to an embodiment of the present disclosure, when the motion of the user terminal device by the user's gesture is detected, determining the user's gesture based on the movement of the user terminal device , when it is determined that the user's gesture is a preset gesture, receiving, from a display device, information on an object included in an image displayed on the display device, information on the object, and a plurality of applications related to the object and displaying a screen provided by the selected application based on the information on the object when a user command for selecting one of the plurality of applications is received.

In addition, the displaying of the information may include displaying information on the object, and when a user command for selecting the object is received, display information on the object and information on a plurality of applications related to the object. can

In addition, the displaying of the information includes displaying an image in which a UI corresponding to the object is superimposed on the image displayed on the display device, and when a user command for selecting the UI is received, information on the object and Information on a plurality of applications related to the object may be displayed.

And, the displaying of the information may include obtaining information on the number of executions of a plurality of applications based on the user's history information, and displaying information on the plurality of applications based on the information on the number of executions. can

And, the displaying of the information includes receiving, from the display device, a plurality of image frames displayed on the display device, displaying an indicator for selecting one image frame from among the plurality of image frames, and setting the indicator. Through this, when a user command for selecting one image frame from among the plurality of image frames is received, information on an object included in the selected image frame and information on a plurality of applications related to the object may be displayed.

In the control method, when it is determined that the user's gesture is a gesture for executing a specific application, the control method determines an object necessary for executing the function of the specific application from among a plurality of objects included in the image displayed on the display device. and displaying a screen provided by the specific application based on the object.

In addition, the displaying of the screen may include determining an object representing a phone number from among a plurality of objects included in the image displayed on the display device when the specific application is a call application, and based on the phone number, the call application The screen provided by this can be displayed.

The displaying of the screen may include determining an object indicating a location from among a plurality of objects included in the image displayed on the display device when the specific application is a map application, and providing the map application based on the location screen can be displayed.

In addition, the control method includes receiving an image displayed on the display device from the display device when the user's gesture is determined as a gesture for screen capture, and displaying a screen provided by a photo application based on the image It may include further steps.

In addition, the control method includes receiving an audio signal output from the display device from the display device when it is determined that the user's gesture is a gesture for executing a music application, and executing the music application based on the audio signal It may further include the step of executing.

According to various embodiments of the present disclosure as described above, a user terminal device capable of providing additional information related to an image displayed on a display device based on a user's gesture and a control method thereof may be provided.

1 is a diagram schematically illustrating an operation of a user terminal device according to an embodiment of the present disclosure.

2 is a block diagram illustrating a user terminal device according to an embodiment of the present disclosure.

3A is a diagram illustrating a user's gesture according to an embodiment of the present disclosure.

3B is a diagram for explaining an embodiment in which a display device recognizes a user's gesture according to an embodiment of the present disclosure;

4 is a view for explaining an embodiment of displaying information on an object according to an embodiment of the present disclosure.

5 is a view for explaining an embodiment of displaying information about an object on a plurality of areas of a display in different formats according to an embodiment of the present disclosure;

6 is a view for explaining an indicator according to an embodiment of the present disclosure.

7 is a view for explaining an embodiment of displaying information on an object and information on a plurality of applications related to an object according to an embodiment of the present disclosure.

8 is a view for explaining an embodiment of displaying information on a plurality of different applications according to user history information according to an embodiment of the present disclosure.

9 is a diagram for explaining a screen provided by an application using an object according to an embodiment of the present disclosure.

10 is a diagram for explaining a screen provided by a preset application using an object according to an embodiment of the present disclosure.

11 is a diagram for explaining an embodiment of executing a call application according to an embodiment of the present disclosure.

12 is a diagram for explaining an embodiment of executing a photo application according to an embodiment of the present disclosure.

13 is a diagram for explaining an embodiment of executing a map application according to an embodiment of the present disclosure.

14 is a diagram for explaining an embodiment of executing a music application according to an embodiment of the present disclosure.

15 is a detailed block diagram illustrating a user terminal device according to an embodiment of the present disclosure.

16 is a view for explaining a display device according to an embodiment of the present disclosure.

17 is a flowchart illustrating an operation of a user terminal device according to an embodiment of the present disclosure.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. However, it is not intended to limit the technology described in this document to specific embodiments, and it should be understood that various modifications, equivalents, and/or alternatives of the embodiments of this document are included. . In connection with the description of the drawings, like reference numerals may be used for like components.

In this document, expressions such as "has," "may have," "includes," or "may include" refer to the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this document, expressions such as "A or B," "at least one of A or/and B," or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component) When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

As used herein, the expression "configured to (or configured to)" depends on the context, for example, "suitable for," "having the capacity to ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a processor configured (or configured to perform) A, B, and C” refers to a dedicated processor (eg, an embedded processor) for performing the operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

The user terminal device 100 according to an embodiment of the present disclosure may transmit/receive various data by performing communication with the display device 200 . To this end, the user terminal device 100 may be communicatively connected to the display device 200 through various communication methods such as Wi-Fi and Bluetooth.

For example, when the user's gesture is recognized, the user terminal device 100 may transmit a signal requesting information on an object included in an image displayed on the display device 200 to the display device 200 . Specifically, when a motion of the user terminal device 100 is detected through a sensor, the user terminal device 100 determines a user's gesture based on the motion of the user terminal device 100, and the user's gesture is a preset gesture. If it is determined that , a signal requesting information on an object included in an image displayed on the display apparatus 200 may be transmitted to the display apparatus 200 .

In this case, the display apparatus 200 may analyze the image displayed on the display apparatus 200 to identify an object included in the image. Here, the identified object may be various objects such as text, numbers, faces, and marks included in the image.

Specifically, when a signal for requesting information on an object is received from the user terminal device 100 , the display apparatus 200 applies an object recognition algorithm to the image displayed on the display apparatus 200 to obtain an object included in the image. can be identified. Here, the object recognition algorithm may be various algorithms such as a face recognition algorithm, a text recognition algorithm, and a number recognition algorithm. Also, the display apparatus 200 may identify an object included in an image using an artificial intelligence model. Specifically, when a signal for requesting information on an object is received from the user terminal device 100, the display apparatus 200 uses a convolutional layer for extracting image feature information and based on the extracted feature information. The object included in the image can be identified through the CNN Convolutional Neural Network model including the fully-connected layer, which is trained to identify the object included in the image. Meanwhile, this is an embodiment, and the display apparatus 200 may identify an object included in an image through various methods such as an edge detection algorithm.

In addition, the display apparatus 200 may transmit information on the identified object to the user terminal apparatus 100 . For example, when the car A is identified in the image, the display apparatus 200 may transmit information about the car A to the user terminal device 100 .

Accordingly, the user terminal device 100 may display information on an object included in the image displayed on the display device 200 . Specifically, when information on an object is received from the display apparatus 200 , the user terminal device 100 performs a web search using the object as a keyword through the web search application installed in the user terminal device 100 , Information obtained as a result of the search may be displayed on the display of the user terminal device 100 .

For example, as shown in FIG. 1 , if the object identified by the display device 200 is a car A, the user terminal device 100 receives information about the car A from the display device 200, and the car A A web search result as a keyword may be displayed on the display of the user terminal device 100 .

Accordingly, the user may be provided with information on the object included in the image displayed on the display apparatus 200 . In particular, since information about an object is transmitted from the display device 200 to the user terminal device 100 by a user's gesture, when a user is interested in an image being played, information about the object can be easily and quickly provided. .

Referring to FIG. 2 , the user terminal device 100 according to an embodiment of the present disclosure may include a communication unit 110 , a sensor 120 , a display 130 , and a processor 140 .

The communication unit 110 may transmit/receive various data by performing communication with various external devices.

For example, the communication unit 110 may communicate with the display apparatus 200 . Specifically, the communication unit 110 is connected to the display device 200 through various communication methods such as Wi-Fi and Bluetooth, and may transmit/receive various data to and from the display device 200 . To this end, the communication unit 110 may include a Wi-Fi module and a Bluetooth module.

For example, the communication unit 110 may transmit information about the user's gesture to the display apparatus 200 . Also, the communication unit 110 may transmit a signal requesting information on an object included in an image displayed on the display apparatus 200 to the display apparatus 200 , and from the display apparatus 200 to the display apparatus 200 . Information on an object included in the displayed image may be received. Also, the communication unit 110 may receive, from the display apparatus 200 , information about an image displayed on the display apparatus 200 . Here, the information about the image may include at least one image frame.

Meanwhile, the communication unit 110 may communicate with the server to transmit/receive various data. Specifically, the communication unit 110 communicates with the server through a mobile communication network such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc., or connects to a Wi-Fi network. Through a communication connection with the server, various data can be transmitted and received with the server. To this end, the communication unit 110 may further include a mobile communication module.

For example, when a user command for executing an application stored in the user terminal device 100 is received, the communication unit 110 is a server that provides a service through the corresponding application, and a signal for requesting information on an image provided by the application. can be sent to the server. In addition, the communication unit 110 may receive information about an image provided by the corresponding application from the server. For example, if the application is an application for web search, the communication unit 110 may receive information about an image including a page for web search from the server.

The sensor 120 may detect the movement of the user terminal device 100 by the user's gesture. To this end, the sensor 120 may include various sensors capable of detecting the movement of the user terminal device 100 , such as an acceleration sensor and a gyro sensor.

Here, the acceleration sensor is a sensor that detects a change in the speed of the user terminal device 100 per unit time. For example, the acceleration sensor may be a 3-axis acceleration sensor that measures the acceleration of the user terminal device 100 in the x-axis, y-axis, and z-axis based on the movement of the user terminal device 100 . In this case, the processor 140 determines that the user terminal device 100 is based on the gravitational acceleration based on the acceleration in the x-axis, y-axis, and z-axis of the user terminal device 100 sensed by the sensor 120 . It is possible to determine the tilted angle, and based on the amount of acceleration change in the x-axis, y-axis, and z-axis of the user terminal device 100 sensed by the sensor 120 for a predetermined time, movement can be judged. Accordingly, the processor 140 may determine the user's gesture based on the movement of the user terminal device 100 and the change in the angle of the user terminal device 100 .

In addition, the gyro sensor is a sensor that detects the angular velocity of the user terminal device 100 . For example, the gyro sensor may be a micro electro mechanical systems (MEMS) gyro sensor that senses the angular velocity of the user terminal device 100 based on a Coriolis force measured based on the rotation of the user terminal device 100 . In this case, the processor 140 may determine the degree of rotation of the user terminal device 100 based on the amount of change in the angular velocity of the user terminal device 100 sensed by the sensor 120 for a certain period of time, and from this, gestures can be judged.

Meanwhile, the above-described sensor is an embodiment, and the sensor 120 may be implemented as various sensors capable of detecting the movement of the user terminal device. As an example, the sensor 120 may further include a geomagnetic sensor capable of measuring a magnetic field that changes according to the movement of the user terminal device 100 .

The display 130 may display various images. Here, the image is a concept including at least one of a still image or a moving image, and the display 130 may display various images such as multimedia content and game content. Also, the display 130 may display various user interfaces (UIs) and icons.

In particular, the display 130 may display information about the object received from the display apparatus 200 . Here, the object is an object included in the image displayed on the display device 200 , and for example, when the car A is included in the image, the display 130 may display information about the car A.

Also, the display 130 may display information about a plurality of applications. Here, the plurality of applications are applications related to an object included in an image displayed on the display device 200 , and specifically, the display 130 displays a plurality of applications that can perform a function using the object included in the image. can do.

Also, the display 130 may display an application execution screen. Here, the application execution screen is a screen provided by an application executed according to a user command, and specifically, may be an image received from a server that provides a service through the corresponding application.

Such a display 130 includes various types of displays, such as Liquid Crystal Display Panel (LCD), Light Emitting Diode (LED), Organic Light Emitting Diodes (OLED), Liquid Crystal on Silicon (LCoS), Digital Light Processing (DLP), and the like. can be implemented as In addition, the display 130 may include a driving circuit, a backlight unit, and the like, which may be implemented in the form of an a-si TFT, a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT).

The processor 140 controls the overall operation of the user terminal device 100 . To this end, the processor 140 may include at least one of a central processing unit (CPU), an application processor (AP), or a communication processor (CP).

The processor 140 may control hardware or software components connected to the processor 140 by driving an operating system or an application program, and may perform various data processing and operations. In addition, the processor 140 may load and process commands or data received from at least one of the other components into the volatile memory, and store various data in the non-volatile memory.

Hereinafter, an operation of the processor 140 will be described in detail with reference to FIGS. 3 to 14 .

The processor 140 may detect the movement of the user terminal device 100 by the user's gesture. Specifically, the processor 140 may detect the movement of the user terminal device 100 through the sensor 120 . To this end, the processor 140 may receive sensing data generated by the sensor 120 from the sensor 120 .

For example, when the sensor 120 is implemented as a three-axis acceleration sensor, the processor 140 transmits sensing data including information on the acceleration of the user terminal device 100 in the x-axis, y-axis, and z-axis to the sensor. 120 may be received. In this case, the processor 140 determines that the user terminal device 100 is based on the gravitational acceleration based on the acceleration in the x-axis, y-axis, and z-axis of the user terminal device 100 sensed by the sensor 120 . It is possible to determine the tilted angle, and based on the amount of acceleration change in the x-axis, y-axis, and z-axis of the user terminal device 100 sensed by the sensor 120 for a predetermined time, movement can be judged. Accordingly, the processor 140 may determine the user's gesture based on the movement of the user terminal device 100 and the change in the angle of the user terminal device 100 .

In addition, when the sensor 120 is implemented as a gyro sensor, the processor 140 receives, from the sensor 120 , sensing data obtained by converting the Coriolis force measured based on the rotation of the user terminal device 100 into an electrical signal. can do. In this case, the processor 140 may determine the amount of change in the angular velocity of the user terminal device 100 sensed by the sensor 120 for a predetermined time based on the sensed data, and may determine the user's gesture therefrom.

In addition, the processor 120 determines the movement of the user terminal device 100 , the inclination angle, and the degree of rotation of the user terminal device 100 based on the sensing data received from the acceleration sensor and the sensing data received from the gyro sensor. and may determine the user's gesture from this.

Meanwhile, the above-described sensor is an embodiment, and the processor 140 may determine the user's gesture through various sensors such as a geomagnetic sensor capable of measuring a magnetic field that changes according to the movement of the user terminal device 100 .

When it is determined that the user's gesture is a preset gesture, the processor 140 may transmit a signal requesting information on an object included in an image displayed on the display apparatus 200 to the display apparatus 200 . To this end, the user terminal device 100 may pre-store information about a preset gesture.

For example, referring to FIG. 3A , the preset gesture may be a gesture of lifting the user terminal device from a horizontal state to a vertical state.

Meanwhile, the user's gesture may be recognized by the display apparatus 200 . For example, referring to FIG. 3B , the display apparatus 200 includes a camera 210 , and the display apparatus 200 may determine a user's gesture based on an image captured by the camera 210 . . Specifically, the display apparatus 200 may identify an object included in the image by applying an object recognition algorithm to an image captured by the camera 210 . In addition, the display apparatus 200 may determine the user's gesture based on a change in an object included in the image. Specifically, the display apparatus 200 determines the movement of the object based on a change in the pixel value of the object included in the image, and determines whether the user's gesture is a preset gesture stored in the display apparatus 200 based on the movement of the object. can be judged

Also, the display apparatus 200 may identify an object included in an image captured by the camera 210 using an artificial intelligence model. Specifically, the display device 200 is a convolutional layer for extracting feature information of an image and a fully-connected layer trained to identify an object included in an image based on the extracted feature information through a CNN model, An object included in an image can be identified. In addition, the display apparatus 200 may determine the movement of the object by tracking the object included in the image, and determine whether the user's gesture is a preset gesture stored in the display apparatus 200 based on the movement of the object. . Meanwhile, this is an embodiment, and the display apparatus 200 may identify an object included in an image through various methods such as an edge detection algorithm.

When a signal requesting information on an object is received from the user terminal device 100 , the display apparatus 200 may transmit information on the object included in the image displayed on the display apparatus 200 to the user terminal apparatus 100 . have. Alternatively, even when the user's gesture is determined to be a preset gesture by the camera 210 , the display apparatus 200 transmits information about an object included in the image displayed on the display apparatus 200 to the user terminal apparatus 100 . can be sent to

To this end, the display apparatus 200 may analyze the object included in the image displayed on the display apparatus 200 to identify the object included in the image. Here, the identified object may be various objects such as text, numbers, faces, and marks included in the image. For example, the text may be a caption included in an image, a shop name on a signboard, and the like, and the number may be a phone number or an account number. In addition, the face may be a human face, as well as an animal face other than a human, such as a dog or a cat. And, the mark may be a brand mark of a car, a brand mark of a bag, a brand mark of clothing, and the like.

Specifically, the display apparatus 200 may identify an object included in the image by applying an object recognition algorithm to the image displayed on the display apparatus 200 . Here, the object recognition algorithm may be various algorithms such as a face recognition algorithm, a text recognition algorithm, and a number recognition algorithm. Also, the display apparatus 200 may identify an object included in an image using an artificial intelligence model. Specifically, the display device 200 is a convolutional layer for extracting feature information of an image and a fully-connected layer trained to identify an object included in an image based on the extracted feature information through a CNN model, An object included in an image can be identified. Meanwhile, as an example, the display device 200 includes a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), and a Bidirectional Recurrent Deep Neural Network (BRDNN). and various artificial intelligence models such as Deep Q-Networks, or edge detection algorithms, etc., to identify objects included in the image.

In addition, the display apparatus 200 may transmit information on the identified object to the user terminal apparatus 100 . For example, the display apparatus 200 transmits information about the car A, the actor B, the actor C, and the cafe D to the user terminal device 100 when the car A, the actor B, the actor C, and the cafe D are identified in the image. can be transmitted

Meanwhile, in the above description, the display apparatus 200 identifies an object included in an image, but the object identification may be performed by the user terminal apparatus 100 . As an example, if it is determined that the user's gesture is a preset gesture based on the sensed data received from the sensor 120 , the processor 140 transmits information about the image displayed on the display apparatus 200 to the display apparatus 200 . A request signal can be transmitted. In addition, when information about an image is received from the display apparatus 200 , the processor 140 may identify an object included in the image through various methods such as the object recognition algorithm or artificial intelligence model described above for the image. Alternatively, when it is determined that the user's gesture is a preset gesture through the camera 210, the display apparatus 200 transmits information about the image displayed on the display apparatus 200 to the user terminal apparatus 100, and the processor ( When information on an image is received from the display apparatus 200 , the 140 may identify an object included in the image through various methods such as the object recognition algorithm or artificial intelligence model described above for the image.

When information on an object is received from the display apparatus 200 , the processor 140 may display information on the object on the display 130 . Alternatively, when an object included in the image is identified as a result of analyzing the image received from the display apparatus 200 , the processor 140 may display information about the object on the display 130 .

For example, if the information on the object includes information on car A, actor B, actor C, and cafe D, the processor 140 displays information on car A, actor B, actor C, and cafe D ( 130) can be indicated. Here, the information on the object may be displayed on the display 130 in a text format, but is not limited thereto.

For example, referring to FIG. 4 , the processor 140 may display an image in which information on an object is superimposed on an image displayed on the display apparatus 200 on the display 130 . To this end, the processor 140 may receive information about an image in which information about an object is superimposed on an image displayed on the display apparatus 200 from the display apparatus 200 . Specifically, when a signal requesting information about an object is received from the user terminal device 100 or it is determined that the user's gesture is a preset gesture through the camera 210, the display apparatus 200 is included in the image. The identified object may be identified, a region in which the identified object is located among the entire region of the image may be determined, and information about an image in which information on the object is superimposed on the determined region may be transmitted to the user terminal device 100 . Alternatively, the processor 140 identifies an object in the image received from the display apparatus 200 , determines a region in which the identified object is located among the entire region of the image, and superimposes information on the object on the determined region. can be generated and displayed on the display 130 .

Meanwhile, the processor 140 may display, on the display 130 , an image in which information about an object is superimposed on an image displayed on the display apparatus 200 , and information about an object listed in text format. For example, referring to FIG. 5 , the processor 140 displays an image in which information about an object is superimposed on a first area of the display 130 , and displays thumbnail information about the object on a second area of the display 130 . and can be displayed by listing in text format. Meanwhile, in FIGS. 4 and 5 , object 1 and object 2 are superimposed and displayed on the image, but according to an embodiment, the processor 140 may superimpose and display the car A, actor B, etc. on the image. .

Meanwhile, the processor 140 receives a plurality of image frames displayed on the display apparatus 200 from the display apparatus 200 , and displays an indicator for selecting one image frame among the plurality of image frames on the display 130 . can do. In this case, when a user command for selecting one image frame among a plurality of image frames is received through the indicator, the processor 140 may receive information about an object included in the selected image frame from the display apparatus 200 . can

Specifically, when it is determined that the user's gesture is a preset gesture based on the sensed data received from the sensor 120 , the processor 140 may display a plurality of image frames displayed on the display apparatus 200 with the display apparatus 200 . You can request a transfer. In addition, the processor 140 may receive a plurality of image frames from the display apparatus 200 . Alternatively, when it is determined that the user's gesture is a preset gesture through the camera 210 , the display apparatus 200 may transmit a plurality of image frames displayed on the display apparatus 200 to the user terminal apparatus 100 .

In this case, as shown in FIG. 6 , the processor 140 displays an indicator 610 for selecting one image frame among a plurality of image frames on the display 130 , and through the indicator 610 , a plurality of It is possible to receive a user command for selecting one image frame from among the image frames of . Here, the user command may be a user's touch input or a touch-and-drag input, but is not limited thereto. In addition, the processor 140 may display information on the object included in the image frame selected according to a user command on the display 130 .

Specifically, the processor 140 may identify an object included in the selected image frame using the above-described object recognition algorithm or artificial intelligence model, and display information on the identified object on the display 130 . Alternatively, the processor 140 may transmit information on the selected image frame to the display apparatus 200 , receive information about an object included in the selected image frame from the display apparatus 200 , and display it on the display 130 . have.

Accordingly, the user can easily select an image frame including the object of interest and check the object of interest.

Meanwhile, the processor 140 may display information on a plurality of applications related to the object along with the information on the object on the display 130 . Here, the object-related application is an application capable of providing information or performing a function using the identified object. For example, if the identified object is an actor, an application capable of providing profile information of the actor, the actor It may be an application that can provide information about a drama or movie that has appeared, and if the identified object is a product such as a car, an application that can provide specification information about the product, information about a shop that sells the product It may be an application that provides , and if the identified object is a store such as a cafe, it may be an application capable of providing a home page screen of the store.

Specifically, the processor 140 displays information on the object on the display 130 , and when a user command for selecting an object is received, the processor 140 displays information on the object and information on a plurality of applications related to the object on the display 130 . ) can be displayed. For example, referring to FIG. 7 , the processor 140 displays, on the display 130 , an image in which a user interface (UI) corresponding to the object is superimposed on the image displayed on the display apparatus 200 in an area where the object is located. When a user command for selecting a UI is received, information on an object corresponding to the selected UI and information on a plurality of applications related to an object corresponding to the selected UI may be displayed on the display 130 . As shown in FIG. 7 , when car A is selected, the processor 140 may provide information on car A or perform a function by using car A, an application for web search, and a specification search for An application, an application for sharing information, and an application for shopping may be displayed.

Meanwhile, the plurality of applications displayed here may be changed based on the user's history information. Here, the user's history information includes information on the number of executions of a plurality of applications, and may be updated whenever the user executes the application. Specifically, when a user command for selecting an object is received, the processor 140 may acquire information on the number of executions of a plurality of applications based on the user's history information. In addition, the processor 140 may display information on a plurality of applications listed in the order of the applications having the highest number of executions on the display 130 . If the user's history information is not stored, the processor 140 may display information on a plurality of applications on the display 130 according to a default order.

As an example, referring to FIG. 8 , when car A is selected in the absence of user history information, the processor 140 displays a plurality of applications related to car A on the display 130 in a default order, and the user If car A is selected in a state in which there is history information of , information on a plurality of applications listed in the order of the applications having the highest number of executions among the plurality of applications may be displayed on the display 130 . Accordingly, the user may be provided with information on a plurality of personalized applications.

When a user command for selecting one of a plurality of applications displayed on the display 130 is received, the processor 140 may display a screen provided by the selected application on the display 130 based on the information on the object. . Specifically, when a user command for selecting one of a plurality of applications displayed on the display 130 is received, the processor 130 executes the selected application and is a server that provides a service through the corresponding application, the identified object information can be transmitted. In addition, the processor 140 may receive an image provided by the server from the server based on the information on the identified object. For example, as shown in FIG. 9 , when car A is selected and an application for web search is selected, the processor 140 requests a web search service server providing a web search service to perform a web search for car A as a keyword. , and can receive information about a web searched page with car A as a keyword from the server. In addition, the processor 140 may display a screen provided by the application on the display 130 based on the information on the image received from the server.

Meanwhile, the processor 140 may execute a preset application when information on the identified object is selected. Specifically, the processor 140 displays information on the identified object on the display 130 , and when a user command for selecting an object is received, a screen provided by a preset application using the selected object is displayed on the display 130 . can be displayed in For example, referring to FIG. 10 , the processor 140 displays a plurality of objects on the display 130 , and when a user command for selecting one of the plurality of objects is received, a preset application is provided based on the selected object. screen can be displayed. As shown in FIG. 10 , if the preset application is an application for web search, the processor 130 may display a web search page using the selected object as a keyword on the display 130 .

Meanwhile, in the above description, when the user's gesture is a preset gesture, the user terminal device 100 displays information on an object and information on a plurality of applications related to the object. However, according to an embodiment, when it is determined that the user's gesture is a gesture for executing a specific application, the user terminal device 100 may display a screen provided by the specific application based on the specific object included in the image. Hereinafter, it will be described with reference to FIGS. 11 to 14 .

The processor 140 may determine whether the user's gesture is a gesture for executing a specific application based on the sensed data received from the sensor 120 . Specifically, the processor 140 may compare the user's gesture with information on a plurality of pre-stored gestures to determine whether the user's gesture is a gesture for executing a specific application.

In addition, when the user's gesture is determined to be a gesture for executing a specific application, the processor 140 selects an object necessary for executing the function of the specific application from among a plurality of objects included in the image displayed on the display device 200 . can judge To this end, the user terminal device 100 may match and store information on at least one application capable of performing a function using the object for each object. For example, when the user's gesture is determined to be a gesture for executing a call application, the processor 140 determines an object representing a phone number from among a plurality of objects included in the image, and the user's gesture determines the execution of the map application. When it is determined as a gesture for , it is possible to determine an object indicating a position from among a plurality of objects included in the image. Here, the object indicating the location may be a name of a region as well as a business name of a restaurant or the like.

In addition, the processor 140 may identify an object required to execute a function of a specific application from the image. Specifically, as described above, the processor 140 may identify an object required for execution of a function of a specific application from an image using an object recognition algorithm or an artificial intelligence model. To this end, the processor 140 may receive information about an image displayed on the display apparatus 200 from the display apparatus 200 .

Meanwhile, the identification of the object may be performed by the display apparatus 200 . For example, when the user's gesture is determined to be a gesture for executing a specific application through the camera 210 , the display device 200 may display a function of a specific application among a plurality of objects included in an image displayed on the display device 200 . It is possible to identify an object necessary for the execution of , and transmit information on the identified object to the user terminal device 100 .

The processor 140 may display a screen provided by a specific application on the display 130 based on the identified object. Specifically, the processor 140 may execute a specific application corresponding to the user's gesture, and display an application execution screen based on the identified object on the display 130 . As an example, referring to FIG. 11 , when the user's gesture is a gesture for executing a call application, the processor 140 identifies an object representing a phone number from among a plurality of objects included in an image, and assigns the number to the identified phone number. Based on the screen provided by the call application may be displayed on the display 130 .

Alternatively, referring to FIG. 12 , when the user's gesture is determined to be a gesture for screen capture, the processor 140 receives an image displayed on the display device from the display device 200 and receives the image displayed on the display device from the display device 200 . A screen provided by the photo application may be displayed on the display 130 based on the image. Specifically, when the user's gesture is determined as a gesture for screen capture, the processor 140 transmits a signal requesting transmission of the image displayed on the display apparatus 200 to the display apparatus 200 , and the display apparatus 200 ), the received image may be stored, and the stored image may be displayed on the display 130 through a photo application.

In addition, referring to FIG. 13 , when the user's gesture is a gesture for executing a map application, the processor 140 identifies an object indicating a location, such as a region name or a business name, among a plurality of objects included in the image, A screen provided by the map application may be displayed on the display 130 based on the identified object. Specifically, when the user's gesture is a gesture for executing the map application, the processor 140 executes the map application, transmits information on the region name, etc. identified as a server providing a service through the map application, and the server A screen including map information for a corresponding area may be received from the display 130 and displayed on the display 130 .

Also, when the user's gesture is determined to be a gesture for executing a music application, the processor 140 receives an audio signal output from the display apparatus 200 from the display apparatus 200, and based on the audio signal, the display apparatus At 200 , the title of the output music may be determined. Specifically, the processor 140 receives the audio signal output from the display device 200 through a speaker, and analyzes the frequency, tempo, energy, etc. of the audio signal, so that the title of the music output from the display device 200 is displayed. can judge In addition, the processor 140 may execute a music application based on the title of the music to output the music being output on the display device 200 .

As described above, user convenience can be improved in that the user terminal device 100 can execute a specific object using a specific object only with a user's gesture.

15 , the user terminal device 100 according to an embodiment of the present disclosure includes a communication unit 110 , a sensor 120 , a display 130 , a storage unit 140 , an input unit 150 , and a microphone 160 . ) and a processor 170 . Hereinafter, portions overlapping with the above description will be omitted or abbreviated.

The communication unit 110 may communicate with the display apparatus 200 . In particular, the communication unit 110 may communicate with the display apparatus 200 through account information of the user terminal apparatus 100 . For example, when the account information of the user terminal device 100 and the account information of the display device 200 are the same, the communication unit 110 may communicate with the display device 200 through a server that provides an account service. . Accordingly, the communication unit 110 may communicate with the display apparatus 200 to receive information about the object.

The storage unit 140 may store an operating system (OS) for controlling overall operations of the components of the user terminal device 100 and commands or data related to the components of the user terminal device 100 .

Accordingly, the processor 170 may control a plurality of hardware or software components of the user terminal device 100 using various commands or data stored in the storage unit 140 , and at least one of the other components. The command or data received from the volatile memory may be loaded and processed, and various data may be stored in the non-volatile memory.

In particular, the storage 140 may store information on a plurality of gestures. For example, the storage unit 140 may store information on a gesture for displaying information on an object included in an image displayed on the display device 200 and information on a plurality of applications related to the object, and information on a specific application for executing a specific application. Information about the gesture can be stored.

Also, the storage 140 may store information on a plurality of applications matched to a plurality of objects. Accordingly, when a user command for selecting one of a plurality of objects included in the image is received, the processor 170 provides information on a plurality of applications related to the selected object based on information on a plurality of applications matched with the plurality of objects. information may be displayed on the display 130 . In addition, when the user's gesture is determined to be a gesture for executing a specific application, the processor 140 determines a specific object matched to a specific application based on information on a plurality of applications matched to the plurality of objects, A specific object can be identified in an image.

Also, the storage 140 may store information about an object recognition algorithm or artificial intelligence model capable of identifying an object included in an image.

The input unit 150 may receive various user commands. The processor 170 may execute a function corresponding to a user command input through the input unit 150 .

For example, the input unit 150 may receive a drag input based on a user's gesture. In this case, the processor 140 determines the user's gesture based on the drag input, and when it is determined that the user's gesture is a preset gesture, information on the object included in the image displayed on the display apparatus 200 and the object Information on a plurality of applications related to may be displayed on the display 130 . Alternatively, when the user's gesture is determined to be a gesture for a specific application, the processor 140 may display a screen provided by the specific application on the display 130 based on the object included in the image displayed on the display device 200 . can For example, when the user inputs a drag input for drawing L into the input unit 150 , the processor 150 executes the map application and displays a screen provided by the map application based on the region name included in the image on the display 130 . ) can be displayed.

To this end, the input unit 260 may be implemented as an input panel. The input panel may be implemented in the form of a touch pad or a keypad including various function keys, number keys, special keys, character keys, or the like, or a touch screen method.

The microphone 160 may receive a user's voice. Here, the user's voice may be a voice for executing a specific function of the user terminal device 100 . When the user's voice is received through the microphone 160 , the processor 170 may analyze the user's voice through a speech to text (STT) algorithm and perform a function corresponding to the user's voice.

For example, when a preset voice is received through the microphone 160 , the processor 170 displays information on an object included in an image displayed on the display device 200 and information on a plurality of applications related to the object. (130) can be indicated.

A speaker (not shown) may output various audio. For example, a speaker (not shown) may output various sounds provided by a music application.

Referring to FIG. 16 , the display apparatus 200 according to an embodiment of the present disclosure includes a camera 210 , a communication unit 220 , a display 230 , a storage unit 240 , an input unit 250 , and a microphone 260 . , a speaker 270 , a signal processing unit 280 , and a processor 290 .

The camera 210 may generate an image by photographing a subject. For example, the camera 210 may generate an image including the user by photographing the user in front of the display apparatus 200 . Accordingly, the processor 290 may analyze the plurality of images captured by the camera 210 to determine the user's gesture.

As such, the camera 210 may be disposed on a panel above the center of the display device 200 , but is not limited thereto, and may be disposed at various locations such as a panel below the center of the display device 200 .

Meanwhile, an image captured by the camera 170 may be stored in the storage unit 240 .

The communication unit 220 may communicate with an external device to transmit/receive various data. For example, the communication unit 220 may communicate with a server through a local area network (LAN), an Internet network, and a mobile communication network, as well as a mobile communication network, BT (Bluetooth), BLE (Bluetooth) Low Energy), WI-FI (Wireless Fidelity), Zigbee, NFC, etc. may communicate with the user terminal device 100 through various communication methods.

To this end, the communication unit 220 may include various communication modules for performing network communication. For example, the communication unit 220 may include a Bluetooth module, a Wi-Fi module, a wireless communication module, and the like.

* Meanwhile, the communication unit 220 may communicate with an external device to receive broadcast content (or broadcast signal). Here, the broadcast content may include at least one of video, audio, and additional data (eg, EPG). To this end, the communication unit 220 may include a tuner (not shown), a demodulator (not shown), an equalizer (not shown), and the like.

The storage unit 240 may store an operating system (OS) for controlling overall operations of the components of the display apparatus 200 and commands or data related to components of the display apparatus 200 .

Accordingly, the processor 290 may control a plurality of hardware or software components of the display apparatus 200 by using various commands or data stored in the storage unit 240 , and use at least one of the other components. The received command or data may be loaded into the volatile memory for processing, and various data may be stored in the non-volatile memory.

In particular, the storage unit 240 may store information on a plurality of gestures. For example, the storage unit 240 may include information on a gesture for transmitting information on an object included in an image displayed on the display device 200 to the user terminal device 100 and a specific information required for executing a function of a specific application. Information about a gesture for transmitting information about an object to the user terminal device 100 may be stored.

Also, the storage unit 240 may store information about an object recognition algorithm or artificial intelligence model capable of identifying an object included in an image.

The input unit 250 may receive various user commands. The processor 290 may execute a function corresponding to a user command input through the input unit 250 .

For example, the input unit 250 may receive a user command for transmitting information on an object. In this case, the processor 290 may transmit information on the object included in the image displayed on the display 230 to the user terminal device 100 .

To this end, the input unit 250 may be implemented as an input panel. The input panel may be implemented in the form of a touch pad or a keypad including various function keys, number keys, special keys, character keys, or the like, or a touch screen method.

The microphone 260 may receive a user's voice. Here, the user's voice may be a voice for executing a specific function of the display apparatus 200 . When a user's voice is received through the microphone 260 , the processor 290 may analyze the user's voice through a speech to text (STT) algorithm and perform a function corresponding to the user's voice.

For example, when a user voice requesting transmission of information on an object is received through the microphone 260 , the processor 290 transmits information on the object included in the image displayed on the display 230 to the user terminal device 100 . can be sent to

The speaker 270 may output various sounds. For example, the speaker 270 may output sound of broadcast content.

The signal processing unit 280 performs signal processing on the content received through the communication unit 220 . Specifically, the signal processing unit 280 may perform operations such as decoding, scaling, and frame rate conversion on an image constituting the content to signal the image in a form that can be output by the display apparatus 200 . In addition, the signal processing unit 280 may perform signal processing such as decoding on the audio constituting the content to process the audio signal in a form that can be output from the speaker 270 .

Accordingly, the display 230 may display an image output from the signal processor 280 , and the speaker 270 may output audio output from the signal processor 290 .

The processor 290 controls the overall operation of the display apparatus 200 . To this end, the processor 290 may include at least one of a central processing unit (CPU), an application processor (AP), or a communication processor (CP).

The processor 290 may control hardware or software components connected to the processor 290 by driving an operating system or an application program, and may perform various data processing and operations. In addition, the processor 290 may load and process commands or data received from at least one of the other components into the volatile memory, and store various data in the non-volatile memory.

The processor 290 may control the communication unit 220 to transmit information on an object included in an image displayed on the display 230 to the user terminal device 100 . Here, information on the object may be obtained through image analysis, as described above. However, the present invention is not limited thereto, and the processor 290 may transmit the metadata to the user terminal device 100 when metadata including information on an object is received from a server providing an image. In this case, the user terminal device 100 may display information on the object based on the information on the object included in the metadata.

When the movement of the user terminal device 100 by the user's gesture is detected through the sensor 120, the user terminal device 100 determines the user's gesture based on the movement of the user terminal device 100 (S1710). can

As an example, when the sensor 120 is implemented as a three-axis acceleration sensor, the user terminal device 100 provides sensing data including information on the acceleration of the user terminal device 100 in the x-axis, y-axis, and z-axis. Based on , it is possible to determine the tilt angle of the user terminal device 100 based on the gravitational acceleration, and the x-axis, y-axis, and z of the user terminal device 100 sensed by the sensor 120 for a predetermined time. The movement of the user terminal device 100 may be determined based on the amount of acceleration change in the axis. In addition, the user terminal device 100 may determine the user's gesture based on the movement of the user terminal device 100 and the change in the angle of the user terminal device 100 .

In addition, when the sensor 120 is implemented as a gyro sensor, the user terminal device 100 converts the Coriolis force measured based on the rotation of the user terminal device 100 into an electrical signal. Based on the sensing data, the user The degree of rotation of the terminal device 100 may be determined, and the user's gesture may be determined based thereon.

When it is determined that the user's gesture is a preset gesture, the user terminal device 100 may receive information about an object included in an image displayed on the display device 200 from the display device 200 ( S1720 ).

Specifically, when it is determined that the user's gesture is a preset gesture, the user terminal device 100 may transmit a signal requesting information on an object included in the image displayed on the display device 200 to the display device 200 . have. To this end, the user terminal device 100 may pre-store information about a preset gesture.

In this case, the display apparatus 200 may transmit information about an object included in the image displayed on the display apparatus 200 to the user terminal apparatus 100 . For example, the display apparatus 200 may transmit an image in which a UI is superimposed on an image displayed on the display apparatus 200 based on information on an object to the user terminal apparatus 100 .

The user terminal device 100 may display information on the object and information on a plurality of applications related to the object on the display ( S1730 ). Here, the object-related application is an application that can provide information or perform a function using the identified object, for example, if the identified object is an actor A, an application that can provide profile information of the actor A; It may be an application that can provide information about a drama or movie in which actor A appeared.

When a user command for selecting one of a plurality of applications is received, the user terminal device 100 may display a screen provided by the selected application on the display based on the information on the object ( S1740 ).

Specifically, when one application among a plurality of applications is selected, the user terminal device 100 may display a screen provided by the application on the display using the identified object. For example, when the object is car A and an application for web search is selected, the user terminal device 100 transmits a signal requesting a web search for car A as a keyword to a server providing a web search service, and from the server It is possible to receive information about a page searched for on the web with the keyword car A. In addition, the user terminal device 100 may display a screen provided by the application on the display based on the information about the image received from the server.

Meanwhile, the above-described methods according to various embodiments of the present disclosure may be implemented in the form of software or applications that can be installed on an existing display device.

In addition, the above-described methods according to various embodiments of the present disclosure may be implemented only by software upgrade or hardware upgrade of an existing user terminal device.

Meanwhile, a non-transitory computer readable medium in which a program for sequentially executing the method of controlling a user terminal device according to the present invention is stored may be provided.

The non-transitory readable medium refers to a medium that stores data semi-permanently, not a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the various applications or programs described above may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

Claims

In the user terminal device,

communication department;

display;

sensor; and

When the movement of the user terminal device by the user's gesture is detected through the sensor, determining the user's gesture based on the movement of the user terminal device,

When it is determined that the user's gesture is a preset gesture, information on an object included in an image displayed on the display device is received from the display device through the communication unit, information on the object, and a plurality of objects related to the object information about the application of the display is displayed,

and a processor configured to display a screen provided by the selected application on the display based on the information on the object when a user command for selecting one of the plurality of applications is received.
According to claim 1,

The processor is

Displaying information on the object on the display, and when a user command for selecting the object is received, information on the object and information on a plurality of applications related to the object are displayed on the display, a user terminal device .
According to claim 1,

The processor is

An image in which a UI corresponding to the object is superimposed on the image displayed on the display device is displayed on the display, and when a user command for selecting the UI is received, information on the object and a plurality of applications related to the object For displaying information on the display, the user terminal device.
According to claim 1,

The processor is

Obtaining information on the number of executions of a plurality of applications based on the user's history information, and displaying information on the plurality of applications on the display based on the information on the number of executions.
According to claim 1,

The processor is

Receive, from the display device, a plurality of image frames displayed on the display device, and display an indicator for selecting one image frame among the plurality of image frames on the display, and through the indicator, the plurality of image frames When a user command for selecting one of the image frames is received, information on an object included in the selected image frame and information on a plurality of applications related to the object are displayed on the display.
According to claim 1,

The processor is

When it is determined that the user's gesture is a gesture for executing a specific application,

A user terminal device for determining an object necessary for executing a function of the specific application from among a plurality of objects included in the image displayed on the display device, and displaying a screen provided by the specific application on the display based on the object .
7. The method of claim 6,

The processor is

When the specific application is a call application, determining an object representing a phone number from among a plurality of objects included in the image displayed on the display device, and displaying a screen provided by the call application on the display based on the phone number , a user terminal device.
7. The method of claim 6,

The processor is

When the specific application is a map application, the user determines an object indicating a location from among a plurality of objects included in the image displayed on the display device, and displays a screen provided by the map application on the display based on the location terminal device.
According to claim 1,

The processor is

When it is determined that the user's gesture is a gesture for screen capture, the user terminal device receives an image displayed on the display device from the display device, and displays a screen provided by a photo application on the display based on the image. .
According to claim 1,

The processor is

When it is determined that the user's gesture is a gesture for executing a music application, an audio signal output from the display device is received from the display device, and a title of music output from the display device is determined based on the audio signal and executing the music application based on the title of the music.
A method for controlling a user terminal device, comprising:

determining the user's gesture based on the movement of the user terminal device when the motion of the user terminal device is detected by the user's gesture;

receiving, from a display device, information on an object included in an image displayed on the display device when the user's gesture is a preset gesture;

displaying information on the object and information on a plurality of applications related to the object; and

When a user command for selecting one of the plurality of applications is received, displaying a screen provided by the selected application based on the information on the object;
12. The method of claim 11,

The step of displaying the information is

Displaying information on the object, and when a user command for selecting the object is received, displaying information on the object and information on a plurality of applications related to the object.
12. The method of claim 11,

The step of displaying the information is

An image in which a UI corresponding to the object is superimposed is displayed on the image displayed on the display device, and when a user command for selecting the UI is received, information on the object and information on a plurality of applications related to the object Displaying, a control method of a user terminal device.
12. The method of claim 11,

The step of displaying the information is

A method of controlling a user terminal device, obtaining information on the number of executions of a plurality of applications based on the user's history information, and displaying information on the plurality of applications based on the information on the number of executions.
12. The method of claim 11,

The step of displaying the information is

Receive, from the display device, a plurality of image frames displayed on the display device, display an indicator for selecting one image frame from among the plurality of image frames, and display an indicator for selecting one image frame from among the plurality of image frames through the indicator. When a user command for selecting an image frame is received, information on an object included in the selected image frame and information on a plurality of applications related to the object are displayed.