KR102161699B1 - Display apparatus and Method for controlling display apparatus thereof - Google Patents

Abstract

A display device and a control method thereof are provided. The method of controlling the display device includes receiving motion information of the mobile terminal from the mobile terminal, photographing the mobile terminal to obtain a mobile terminal image, and converting the received movement information into a screen coordinate system aligned with the screen of the display device. , Corrects motion information converted to a screen coordinate system based on the mobile terminal image, and controls the display device using the corrected motion information.

Description

TECHNICAL FIELD [0001] Display apparatus and method for controlling display apparatus thereof

The present invention relates to a display device and a control method thereof, and more particularly, to a display device for controlling various functions of a display device using motion information of an external portable terminal, and a control method thereof.

In recent years, in order to more intuitively control the display device, a method for controlling the function of the display device using the movement of the user has been developed. For example, the mobile terminal may detect motion information according to a user motion and transmit the detected motion information to an external display device. In addition, the display device controls various functions (eg, pointing movement, etc.) of the display device by using the transmitted motion information. As another example, the display device controls various functions of the display device based on the captured motion information by photographing the movement of the mobile terminal.

However, as in the prior art, in the case of a method of transmitting motion information by a portable terminal, only the relative motion of the portable terminal is sensed, and it is difficult to accurately recognize a point actually indicated by the portable terminal. In addition, in the case of a method of photographing a portable terminal by a display device, it is difficult to accurately detect a three-dimensional motion of the portable terminal.

The present invention was conceived to solve the above-described problem, and an object of the present invention is to correct the motion information received from the mobile terminal using the captured mobile terminal image, so that the display device is more accurate and intuitive using the mobile terminal. A display device capable of controlling a display device and a method for controlling the same are provided.

According to an embodiment of the present invention for achieving the above object, a method for controlling a display device includes the steps of: receiving motion information of the mobile terminal from a mobile terminal, photographing the mobile terminal to obtain a mobile terminal image; Converting the received motion information into a screen coordinate system aligned with a screen of the display device; Correcting motion information converted to the screen coordinate system based on the mobile terminal image; And controlling the display device using the corrected motion information.

And, the motion information received from the mobile terminal is a value measured based on a mobile terminal coordinate system aligned with the mobile terminal, and the converting step includes geomagnetic motion information measured based on the mobile terminal coordinate system. Converting into a global coordinate system aligned based on a direction; And converting the motion information converted to the global coordinate system into the screen coordinate system.

In addition, the converting may further include converting motion information converted into the screen coordinate system into the camera coordinate system when the camera coordinate system aligned with the camera photographing the mobile terminal and the screen coordinate system are different from each other; can do,

In addition, the conversion parameter of the screen coordinate system may be calculated based on the shape of the portable terminal when an initialization input is input from the user while the portable terminal is aligned in the screen direction of the display device.

Further, in the step of correcting, the motion information may be corrected by comparing the size of the portable terminal obtained from the portable terminal image with the actual size of the previously obtained portable terminal.

The correcting may include comparing motion information of the portable terminal obtained from the portable terminal image and motion information converted to the screen coordinate system; And when the motion information of the portable terminal acquired from the portable terminal image and the motion information converted to the screen coordinate system do not match, the motion information of the portable terminal obtained from the portable terminal image and the motion converted to the screen coordinate system. It may include; synchronizing information.

In addition, the controlling may include recognizing a three-dimensional motion of the portable terminal using the corrected motion information; And controlling the display device based on the recognized three-dimensional movement.

In addition, the controlling may include: when a plurality of portable terminals exist, detecting a motion pattern corresponding to each of the plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals; And recognizing the plurality of portable terminals using a motion pattern corresponding to each of the plurality of portable terminals.

In addition, in the controlling, the display device may be controlled based on the corrected motion information only when the portable terminal is photographed.

In addition, the controlling may include calculating pointing coordinates based on the corrected motion information; And displaying a pointer on the pointing coordinates.

On the other hand, according to an embodiment of the present invention for achieving the above object, a method for controlling a display device includes: a communication unit for receiving motion information of the portable terminal from the portable terminal; An image acquisition unit for capturing the mobile terminal to obtain a mobile terminal image; And converting the received motion information into a screen coordinate system aligned with the screen of the display device, correcting motion information converted to the screen coordinate system based on the mobile terminal image, and using the corrected motion information And a control unit for controlling the display device.

In addition, the motion information received from the mobile terminal is a value measured based on a mobile terminal coordinate system aligned with the mobile terminal, and the control unit determines the geomagnetic direction with the movement information measured based on the mobile terminal coordinate system. It converts into a global coordinate system aligned with a reference, and converts motion information converted into the global coordinate system into the screen coordinate system.

In addition, the controller may convert motion information converted into the screen coordinate system into the camera coordinate system when the camera coordinate system aligned with the camera photographing the mobile terminal and the screen coordinate system are different.

In addition, the conversion parameter of the screen coordinate system may be calculated based on the shape of the portable terminal when an initialization input is input from the user while the portable terminal is aligned in the screen direction of the display device.

In addition, the controller may correct the motion information by comparing the size of the portable terminal obtained from the portable terminal image with the actual size of the previously obtained portable terminal.

In addition, the controller compares the motion information of the portable terminal obtained from the portable terminal image with the motion information converted to the screen coordinate system, and the motion information of the portable terminal obtained from the portable terminal image and the screen coordinate system When the converted motion information does not match, the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system may be synchronized.

In addition, the controller may recognize a 3D motion of the portable terminal using the corrected motion information, and control the display device based on the recognized 3D motion.

And, when a plurality of portable terminals exist, the control unit detects a motion pattern corresponding to each of the plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals, and the plurality of portable terminals The plurality of portable terminals may be recognized using a motion pattern corresponding to each.

Further, the controller may control the display device based on the corrected motion information only when the portable terminal is photographed.

The display unit may further include a display unit, wherein the controller calculates pointing coordinates based on the corrected motion information and controls the display unit to display a pointer at the pointing coordinates.

According to various embodiments of the present invention as described above, a user can more accurately and intuitively control a display device using a portable terminal.

1 is a view showing a display system according to an embodiment of the present invention,
2 is a block diagram schematically showing a configuration of a display device according to an embodiment of the present invention;
3 is a block diagram showing in detail the configuration of a display device according to an embodiment of the present invention;
4 is a diagram showing a plurality of coordinate systems according to an embodiment of the present invention;
5A to 5C are diagrams for explaining a coordinate system transformation according to an embodiment of the present invention;
6 to 13 are diagrams illustrating an embodiment of controlling a display device using motion information of a mobile terminal according to various embodiments of the present invention, and
14 is a flowchart illustrating a method of controlling a display device according to an embodiment of the present invention.

Since these embodiments can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiment, it is to be understood to include all conversions, equivalents, or substitutes included in the disclosed spirit and technical scope. In describing the embodiments, when it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by terms. The terms are only used for the purpose of distinguishing one component from another component.

The terms used in the present application are used only to describe specific embodiments, and are not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "comprise" are intended to designate the existence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other It is to be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance the possibility of being excluded.

In an embodiment, the'module' or'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of'modules' or a plurality of'units' are integrated into at least one module except for'modules' or'units' that need to be implemented with specific hardware and implemented as at least one processor (not shown). Can be.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted.

1 is a diagram illustrating a display system 10 according to an embodiment of the present invention. As shown in FIG. 1, the display system 10 includes a display device 100 and a portable terminal 200 for controlling the display device 100. Meanwhile, according to an embodiment of the present invention, the display device 100 may be a smart TV, but this is only an example, and may be various display devices such as a digital TV, a desktop PC, and a notebook PC. In addition, the portable terminal 200 may be a smart phone, but this is only an example, and may be implemented with various portable terminals 200 such as a pointing device, a remote control, and a tablet PC.

First, the mobile terminal 200 detects motion information according to a user's motion. In this case, the portable terminal 200 may detect motion information using at least one of an acceleration sensor, a geomagnetic sensor, and a gyro sensor. Then, the mobile terminal 200 transmits the detected motion information to the display device 100.

The display apparatus 100 receives motion information of the portable terminal 200 from the portable terminal 200. In this case, the display apparatus 100 acquires a portable terminal image by photographing the portable terminal 200 while receiving motion information from the portable terminal 200.

Then, the display device 100 converts the received motion information into a screen coordinate system aligned with the screen of the display device. Specifically, the display apparatus 100 may convert motion information of the mobile terminal coordinate system aligned with respect to the mobile terminal 200 into a global coordinate system aligned with the geomagnetic direction. In addition, the display apparatus 100 may convert motion information converted into a global coordinate system into a screen coordinate system. In addition, when the screen coordinate system and the camera coordinate system aligned based on the camera do not match, the display apparatus 100 may convert motion information corrected by the screen coordinate system into the camera coordinate system. Meanwhile, the above-described conversion parameter of the screen coordinate system may be calculated based on the shape of the photographed mobile terminal 200 when the user inputs an initialization input while the mobile terminal 200 is aligned in the screen direction of the display device. have.

In addition, the display apparatus 100 may correct motion information converted into a screen coordinate system based on the mobile terminal image. In this case, the display apparatus 100 may correct the size and timing of the portable terminal 200 based on the portable terminal image. Specifically, the display apparatus 100 may compensate the motion information by comparing the size of the portable terminal 200 obtained from the portable terminal image with the actual size of the obtained portable terminal 200. In addition, the display device 100 compares the motion information of the portable terminal 200 obtained from the image of the portable terminal 200 with the motion information converted to the screen coordinate system, and the motion information of the portable terminal obtained from the portable terminal image If the motion information converted to the screen coordinate system does not match, the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system may be synchronized.

In addition, the display device 100 may control various functions of the display device 100 based on the corrected motion information. Specifically, the display apparatus 100 may recognize a 3D motion of the mobile terminal 200 using the corrected motion information, and control the display apparatus based on the recognized 3D motion. In addition, when there are a plurality of portable terminals, the display apparatus 100 detects a motion pattern corresponding to each of the plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals, and A plurality of portable terminals may be recognized using a motion pattern corresponding to. In addition, the display apparatus 100 may calculate the pointing coordinates based on the corrected motion information and display the pointer at the pointing coordinates. Other various embodiments will be described in detail later with reference to the drawings.

2 is a block diagram schematically illustrating a configuration of a display device 100 according to an embodiment of the present invention. As shown in FIG. 2, the display apparatus 100 includes a communication unit 110, an image acquisition unit 120, and a control unit 130.

The communication unit 110 communicates with an external portable terminal 200. In this case, the communication unit 110 may receive movement information of the mobile terminal 200 according to the user's movement from the mobile terminal 200. On the other hand, according to an embodiment of the present invention, the communication unit 110 may perform communication with the mobile terminal 200 in a Bluetooth method, but this is only an embodiment, and another communication method (for example, infrared communication Method, Wi-Fi communication method, ZigBee communication method, etc.) to communicate with the mobile terminal 200.

The image acquisition unit 120 captures an image using a camera and processes the captured image to obtain an image. In this case, the image acquisition unit 120 may acquire an image of a mobile terminal including the mobile terminal 200.

The controller 130 controls the overall operation of the display device 100. In particular, the control unit 130 converts the motion information received from the external mobile terminal 200 into a screen coordinate system aligned with the screen of the display device 100, and converts the movement into a screen coordinate system based on the mobile terminal image. The information is corrected, and the display apparatus 100 is controlled using the corrected motion information.

Specifically, the controller 130 controls the communication unit 110 to receive motion information from the mobile terminal 200. In this case, the motion information received from the mobile terminal 200 is a value measured based on the mobile terminal coordinate system aligned with the mobile terminal 200.

In addition, the controller 130 may convert motion information measured based on the mobile terminal coordinate system into a global coordinate system aligned based on a geomagnetic direction, and convert the motion information converted to the global coordinate system back into a screen coordinate system. In this case, the controller 130 may convert the motion information into a different coordinate system through a rotation transformation matrix. Specifically, when an initialization input is input from a user while the mobile terminal 200 is aligned in the screen direction of the display device 100, the controller 130 calculates a conversion parameter based on the shape of the mobile terminal 200. And store the calculated conversion parameters. In addition, the controller 130 may convert motion information of the mobile terminal coordinate system to the screen coordinate system by using the stored conversion parameter.

Meanwhile, when the camera coordinate system aligned with the camera for photographing the mobile terminal 200 and the screen coordinate system are different, the controller 130 may convert motion information converted into the screen coordinate system back to the camera coordinate system.

In addition, the controller 130 may correct the motion information using the actual size and motion time of the portable terminal. Specifically, the controller 130 may compensate the motion information by comparing the size of the portable terminal 200 obtained from the portable terminal image with the actual size of the obtained portable terminal 200. In addition, the controller 130 compares the motion information of the portable terminal 200 obtained from the portable terminal image and the motion information converted to the screen coordinate system, and the motion information of the portable terminal 200 obtained from the portable terminal image and the screen coordinate system If the motion information converted to does not match, the controller 130 may synchronize the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system.

In addition, the controller 130 may control various functions of the display apparatus 100 by using the corrected motion information.

Specifically, the controller 130 may recognize a 3D motion of the mobile terminal 200 using the corrected motion information, and control the display device based on the recognized 3D motion. In addition, when there are a plurality of portable terminals, the controller 130 detects a motion pattern corresponding to each of the plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals, and A plurality of portable terminals can be recognized using a corresponding motion pattern. In addition, the controller 130 may control the display apparatus 100 based on the corrected motion information only when the mobile terminal 200 is photographed. Also, the controller 130 may calculate pointing coordinates based on the corrected motion information, and display a pointer at the pointing coordinates.

With the display device 100 as described above, the user can control the display device more accurately and intuitively using a portable terminal.

Hereinafter, various embodiments of the present invention will be described with reference to FIGS. 3 to 13. 3 is a block diagram showing in detail the configuration of the display device 100 according to an embodiment of the present invention. As shown in FIG. 3, the display device 100 includes a communication unit 110, an image receiving unit 140, an image processing unit 150, a display unit 160, a storage unit 170, an image acquisition unit 120, and It includes an audio output unit 180, an input unit 190, and a control unit 130.

The communication unit 110 is a component that communicates with various types of external devices according to various types of communication methods. The communication unit 110 may include various communication chips such as an infrared communication chip, a Wi-Fi chip, a Bluetooth chip, an NFC chip, and a wireless communication chip. At this time, the WiFi chip, the Bluetooth chip, and the NFC chip perform communication in a WiFi method, a Bluetooth method, and an NFC method, respectively. Among them, the NFC chip refers to a chip that operates in the NFC (Near Field Communication) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, and 2.45 GHz. In the case of using a Wi-Fi chip or a Bluetooth chip, various types of connection information such as SSID and session key are first transmitted and received, and various types of information may be transmitted and received after communication is connected using the same. A wireless communication chip means a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evoloution (LTE).

In particular, the communication unit 110 may receive movement information of the mobile terminal 200 according to the user's movement from the mobile terminal 200. In this case, the motion information received from the mobile terminal 200 may be a value measured based on the mobile terminal coordinate system aligned with the mobile terminal 200.

The image receiving unit 140 receives image content from various sources. Specifically, the image receiving unit 140 may receive broadcast content from an external broadcasting station and may receive image content from an external device (eg, a DVD player, a set-top box, etc.).

The image processing unit 150 is a component that processes image data received by the image receiving unit 140. The image processing unit 150 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on image data.

The display unit 160 displays at least one of a video frame processed by the image processing unit 150 and various screens generated by the graphic processing unit 133 of image data received from the image receiving unit 140. In particular, the display unit 160 may display a pointer based on motion information of the portable terminal 200.

The storage unit 170 stores various modules for driving the display device 100. For example, software including a base module, a sensing module, a communication module, a presentation module, a web browser module, and a service module may be stored in the storage unit 170. In this case, the base module is a basic module that processes signals transmitted from hardware included in the display apparatus 100 and transfers them to an upper layer module. The sensing module is a module that collects information from various sensors, analyzes and manages the collected information, and may include a face recognition module, a voice recognition module, a motion recognition module, an NFC recognition module, and the like. The presentation module is a module for configuring a display screen, and may include a multimedia module for reproducing and outputting multimedia content, and a UI rendering module for performing UI and graphic processing. The communication module is a module for performing communication with the outside. The web browser module refers to a module that accesses a web server by performing web browsing. The service module is a module including various applications for providing various services.

In addition, the storage unit 170 may store a transformation parameter for converting motion information measured based on the mobile terminal coordinate system to the screen coordinate system.

The image acquisition unit 120 captures an image and processes the captured image to obtain an image. In particular, the image acquisition unit 120 may capture an area including the portable terminal 200 using a camera and process the captured image to obtain a portable terminal image.

The audio output unit 180 is a component that outputs not only various audio data processed by an audio processing unit (not shown), but also various notification sounds or voice messages.

The input unit 190 receives various user commands for controlling the display device 100. In this case, the input unit 190 may be implemented with various input devices such as a remote control, a touch screen, a mouse, a voice input unit, a motion input unit, and the like.

As shown in FIG. 3, the control unit 130 includes a RAM 131, a ROM 132, a graphic processing unit 133, a main CPU 134, the first to n interfaces 135-1 to 135-n, It includes a bus 136. At this time, the RAM 131, the ROM 132, the graphic processing unit 133, the main CPU 134, the first to n interfaces 135-1 to 135-n, and the like may be connected to each other through the bus 136. .

The ROM 132 stores an instruction set for booting the system, and the like. When power is supplied by inputting a turn-on command, the main CPU 134 copies the O/S stored in the storage unit 170 to the RAM 131 according to the instruction stored in the ROM 132 and executes the O/S. To boot the system. When booting is complete, the main CPU 134 copies various application programs stored in the storage unit 170 to the RAM 131 and executes the application programs copied to the RAM 131 to perform various operations.

The graphic processing unit 133 generates a screen including various objects such as pointers, icons, images, texts, and the like using an operation unit (not shown) and a rendering unit (not shown). The calculation unit calculates attribute values such as coordinate values, shape, size, color, etc. to be displayed for each object according to the layout of the screen by using the control command received from the input unit. The rendering unit generates screens of various layouts including objects based on the attribute values calculated by the calculation unit. The screen generated by the rendering unit is displayed in the display area of the display unit 160.

The main CPU 134 accesses the storage unit 170 and performs booting using the O/S stored in the storage unit 170. Then, the main CPU 134 performs various operations using various programs, contents, data, etc. stored in the storage unit 170.

The first to nth interfaces 135-1 to 135-n are connected to the various components described above. One of the interfaces may be a network interface that is connected to an external device through a network.

Meanwhile, prior to describing the operation of the control unit 130, the coordinate system disclosed in the present invention will be described in detail with reference to FIG. 3. The mobile terminal coordinate system 410 is a coordinate system aligned based on the mobile terminal, the global coordinate system 420 is a coordinate system aligned based on a geomagnetic direction, and the screen coordinate system 430 is aligned based on the screen of the display device 100 Is a coordinate system, and the camera coordinate system 440 is a coordinate system in which the cameras provided in the display device 100 are aligned in a mood. The virtual coordinate system 450 is a coordinate system in which an image is formed because it is a certain distance away from the focus of the camera, and the display screen coordinate system 460 is a coordinate system whose size is adjusted to express an image of a mobile terminal through the display device 100, and a normalized screen The coordinate system 470 refers to a coordinate system whose size is corrected through an image of a mobile terminal.

Specifically, the controller 130 may control the communication unit 110 to receive motion information from the mobile terminal 200. In this case, the motion information received from the mobile terminal 200 may be a value (x _d , y _d , z _d ) arranged in the mobile terminal coordinate system.

In addition, the control unit 130 may convert motion information based on the coordinate system of the mobile terminal to the screen coordinate system. Specifically, the controller 130 may convert motion information based on a coordinate system of the mobile terminal into a global coordinate system. For example, the mobile terminal coordinate system and the global coordinate system may be different as shown in FIG. 5A.

At this time, the controller 130 expresses the motion information as Euler angles, and converts the motion information based on the mobile terminal's coordinate system to the values of the global coordinate system (x _E , y _E , z _E ) Can be converted.

At this time, φ refers to a pitch angle, θ refers to a roll angle, and ψ refers to a yaw angle.

In addition, the controller 130 converts the motion information converted to the global coordinate system back into values (x _S , y _S , z _S ) of the screen coordinate system shown in FIG. 5B. In this case, the controller 130 may convert motion information converted into a global coordinate system based on a screen coordinate system by using a transformation parameter of a previously stored screen coordinate system. In this case, the conversion parameter of the screen coordinate system will be calculated based on the shape of the photographed portable terminal 200 when the user inputs an initialization input while the portable terminal 200 is aligned in the screen direction of the display device 100. I can.

Meanwhile, in the above-described embodiment, it is assumed that the screen coordinate system and the camera coordinate system coincide. However, this is only an example, and the screen coordinate system and the camera coordinate system do not match, or as shown in FIG. When the camera coordinate system is different from the screen coordinate system by rotation, the controller 130 may finally convert the motion information of the screen coordinate system into the camera coordinate system.

Meanwhile, in the above-described embodiment, it has been described that the motion information of the mobile terminal coordinate system is converted to the global coordinate system and the global coordinate system is converted to the screen coordinate system, but this is only an embodiment, and the motion information of the mobile terminal coordinate system is converted to the screen coordinate system ( Or it can be changed directly to the camera coordinate system).

In addition, the controller 130 may correct motion information converted into a screen coordinate system based on the mobile terminal image.

Specifically, the controller 130 may compensate the motion information by comparing the size of the portable terminal detected from the portable terminal image acquired by the image acquisition unit 120 with the actual size of the previously acquired portable terminal. Specifically, the controller 130 may calculate a normalization parameter by comparing the size of the portable terminal obtained through the portable terminal image with the actual size of the previously obtained portable terminal. In addition, the controller 130 may normalize the motion information received from the mobile terminal 200 by using the normalization parameter.

In addition, the controller 130 may compare motion information of the portable terminal 200 acquired from the portable terminal image and motion information converted to a screen coordinate system. And, if the motion information of the portable terminal 200 obtained from the portable terminal image and the motion information converted to the screen coordinate system do not match, the controller 130 converts the motion information of the portable terminal obtained from the portable terminal image and the screen coordinate system. The converted motion information can be synchronized. Specifically, the portable terminal 200 may generate timestamp information together with motion information and transmit the generated timestamp information to the display device 100. In addition, the controller 130 may generate timestamp information while acquiring an image of the mobile terminal. In addition, when a constant time shifting is detected by comparing the motion pattern converted to the screen coordinate system with the motion pattern of the mobile terminal image, the control unit 130 may correct the timing by the sensed time shifting.

As described above, by transforming the coordinate system and correcting motion information through the image of the portable terminal, the controller 140 can more accurately know the motion information of the portable terminal 200. In particular, the controller 140 can accurately measure the movement of the x, y, and z axes of the mobile terminal 200 through the above-described method.

In addition, the controller 130 may control various functions of the display apparatus 100 using the corrected motion information as described above.

In an embodiment of the present invention, the control unit 130 determines the three-dimensional movement of the mobile terminal 200 using motion information of the mobile terminal 200, and the mobile terminal 200 determines the three-dimensional movement based on the determined three-dimensional movement. Control commands can be sent to. In particular, when a movement in the y-axis direction is detected, the controller 130 may transmit a control command to the portable terminal 200 to perform a function corresponding to the movement in the y-axis direction. For example, as shown in FIG. 6, the movement of the mobile terminal 200 closer to the display device 100 in the y-axis direction (that is, a movement as if throwing the mobile terminal 200 toward the display device 100) When this is detected, the controller 130 may request a file being played by the mobile terminal 200, and the mobile terminal 200 may transmit the file currently being played back to the display device 100.

In addition, when there are a plurality of portable terminals, the controller 130 may detect a motion pattern corresponding to each of the plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals. In addition, the controller 130 may recognize a plurality of portable terminals using a motion pattern corresponding to each of the plurality of portable terminals. Specifically, the controller 130 may distinguish between a plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals. For example, as shown in FIG. 7, the control unit 130 can distinguish between a mobile terminal having a pattern moving along the x-axis, a mobile terminal having a pattern moving along the y-axis, and a mobile terminal having a pattern moving along the z-axis. have. Further, the control unit 130 detects a user face adjacent to each of the mobile terminals using a camera included in the image acquisition unit 120, compares the detected user face with user information stored in the storage unit 170, and You can identify users who use. That is, the face detected near the mobile terminal with the pattern moving along the x-axis is user1, the face detected near the mobile terminal with the pattern moving along the y-axis is user2, and the face detected near the mobile terminal with the pattern moving along the z-axis. When the resultant face is user3, the controller 130 compares the detected face with user information stored in the storage unit 170 to recognize each user. Accordingly, the control unit 130 may individually control the function of the display apparatus 100 using the detected mobile terminals of each user. In addition, a plurality of users can simultaneously play a game through the display device 100.

Also, the controller 130 may control the display apparatus 100 based on the corrected motion information only when the portable terminal is photographed. Specifically, as shown in FIG. 8, only when the portable terminal 200 is photographed by a camera included in the display apparatus 100, the controller 130 uses the movement of the portable terminal 200 to You can do 100. When the mobile terminal 200 is not photographed by a camera provided in the display device 100, the controller 130 performs a function corresponding to the movement of the mobile terminal 200 even when the movement of the mobile terminal 200 is received. I can't. That is, only when motion information is received from the mobile terminal 200 and an image of the mobile terminal is obtained, the controller 130 may perform various functions of the display device 100 (eg, device-to-device connection, application connection, screen AR). Mode, motion control mode, etc.) can be performed.

Further, the controller 130 may control the display 160 to calculate the pointing coordinates based on the corrected motion information and display a pointer at the pointing coordinates. Specifically, as shown in FIG. 9, the controller 130 detects the current position indicated by the mobile terminal 200 based on the mobile terminal image and motion information of the mobile terminal 200, and points the detected position. You can set the origin. In addition, the controller 130 may control the display 160 to display a pointer by calculating the pointing coordinates from the pointing origin based on the motion information of the portable terminal 200. In this case, the control unit 130 may provide various applications using a pointer. For example, the control unit 130 may control a web browser using a touch pad provided in the mobile terminal 200, a pointing operation, a text box pointing, and a text input pad provided in the mobile terminal 200. As another example, the controller 130 may perform various operations such as an operation of selecting an object on a TV screen or obtaining object information using a pointer. In this case, when a plurality of pointers corresponding to a plurality of portable terminals are used, the controller 130 may control the display unit 160 to display each of the plurality of pointers differently. For example, the controller 130 may control the display unit 160 to display a plurality of pointers by varying the color, contrast, and transparency of the pointer.

Meanwhile, in the above-described embodiment, it has been described that the three-dimensional motion is determined using motion information and the three-dimensional motion is corrected using the mobile terminal image. However, this is only an embodiment, and the three-dimensional motion is performed by another method. Can be obtained. Specifically, as shown in FIG. 10, the controller 160 may determine a two-dimensional movement of the mobile terminal 200, that is, movement of the x-axis and z-axis, from the mobile terminal image, and the movement of the mobile terminal 200 Based on the information, the y-axis movement can be determined.

In addition, the controller 130 may control various functions of the display apparatus 100 using the user's device concentration, that is, the user's gaze. Specifically, as shown in FIG. 11, the controller 130 may detect the user's gaze using a camera included in the image acquisition unit 120. When the user's gaze is toward the portable terminal 200, the controller 130 does not control the display apparatus 100 using motion information of the portable terminal 200. In addition, a user can use various applications using the portable terminal 200. However, when the user's gaze is toward the display apparatus 100, the controller 130 displays the pointer on the display unit 160 and moves the pointer using the movement of the portable terminal 200. In this case, the control unit 130 may stop the application operation of the mobile terminal 200 by transmitting an operation stop request to the mobile terminal 200.

In addition, the controller 130 may detect whether a user enters or exits by using motion information of the mobile terminal 200 and an image of the mobile terminal. Specifically, as illustrated in FIG. 12, the controller 130 may detect whether the user is located within the screen area using the mobile terminal image. In addition, in order to reduce the load of vision processing processing, the controller 130 tracks the user's movement line in the screen area using the movement information of the mobile terminal 200, and tracks the user's movement line using the mobile terminal image only when necessary. I can. Accordingly, the controller 130 may perform various application functions (for example, a function of turning on a light in a space in which the user moves) in conjunction with user motion tracking. In addition, in modeling the indoor space based on the 2D camera, the control unit 130 analyzes and learns the motion radius or the movement of the user with the mobile terminal 200, thereby improving the accuracy of 3D indoor space modeling. .

In addition, the controller 130 may identify the portable terminal 200 using the movement pattern of the portable terminal 200 and then authenticate the user with user information of the portable terminal 200. For example, as shown in FIG. 13, when detecting a first portable terminal having a movement pattern in the x-axis direction, the controller 130 authenticates a user using information on USER1 stored in the first portable terminal. When a second portable terminal having a movement pattern in the z-axis direction is detected, the controller 130 may perform user authentication using information on USER2 stored in the second portable terminal. Accordingly, the user can perform user authentication, such as a login operation, using the movement pattern of the portable terminal 200.

Hereinafter, a method of controlling the display apparatus 100 will be described with reference to FIG. 14.

First, the display apparatus 100 receives motion information of the portable terminal 200 from the portable terminal 200, photographs the portable terminal 200, and acquires a portable terminal image (S1410). In this case, the operation of receiving motion information of the mobile terminal 200 and the operation of photographing the mobile terminal 200 may be performed simultaneously, but this is only an embodiment and may be performed sequentially.

Then, the display apparatus 100 converts the received motion information into a screen coordinate system (S1420). Specifically, the display apparatus 100 may convert motion information of the mobile terminal coordinate system aligned with respect to the mobile terminal 200 into a global coordinate system aligned with the geomagnetic direction. In addition, the display apparatus 100 may convert motion information converted into a global coordinate system into a screen coordinate system using a transformation parameter of the screen coordinate system.

Then, the display apparatus 100 corrects the motion information corrected in the screen coordinate system based on the mobile terminal image (S1430). Specifically, the display apparatus 100 may compensate the motion information by comparing the size of the portable terminal 200 obtained from the portable terminal image with the actual size of the obtained portable terminal 200. In addition, the display device 100 compares the motion information of the portable terminal 200 obtained from the image of the portable terminal 200 with the motion information converted to the screen coordinate system, and the motion information of the portable terminal obtained from the portable terminal image If the motion information converted to the screen coordinate system does not match, the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system may be synchronized.

Then, the display device 100 controls the function of the display device 100 by using the corrected motion information (S1440). Specifically, the display apparatus 100 can control various functions as illustrated in FIGS. 6 to 13 by using the corrected motion information.

By the method of controlling the display device as described above, the user can control the display device more accurately and intuitively using the portable terminal.

Meanwhile, in the above-described embodiment, the display device 100 receives motion information from the portable terminal 200, converts the received motion information into a screen coordinate system aligned with the screen of the display device, and captures the image of the portable terminal. It has been described that the motion information converted to the screen coordinate system is corrected based on, but this is only an example, and an embodiment in which the mobile terminal 200 directly corrects the motion information may also include the technical idea of the present invention.

This will be described with reference to FIG. 15. First, the portable terminal 200 includes a sensing unit 210, a communication unit 220, and a control unit 230 as shown in FIG. 15.

The detector 210 detects motion information of the portable terminal 200 using a plurality of sensors. Specifically, the sensing unit 210 may detect motion information using at least one of a speed sensor, a geomagnetic sensor, and a gyro sensor.

The communication unit 220 communicates with an external display device 100. In particular, the communication unit 220 may receive a mobile terminal image from the display device 100 and transmit corrected motion information to the display device 100.

The controller 230 controls the overall operation of the mobile terminal 200. In particular, the controller 230 calculates a coordinate value based on motion information sensed through the sensing unit 210, and converts the calculated coordinate value into a screen coordinate system aligned with the screen of the display device 100. have. At this time, as described above, the controller 230 may convert the coordinate values in the order of the mobile terminal coordinate system -> global coordinate system -> screen coordinate system.

In addition, the controller 230 controls the communication unit 220 to receive an image of the mobile terminal from the display device 100.

The controller 230 may correct motion information based on the received mobile terminal image. Specifically, the controller 230 may correct the motion information by comparing the size of the portable terminal acquired from the portable terminal image with the actual size of the previously acquired portable terminal. In addition, the controller 230 compares the motion information of the portable terminal obtained from the portable terminal image with the motion information converted to the screen coordinate system, and synchronizes the motion information of the portable terminal obtained from the portable terminal image with the motion information converted to the screen coordinate system. can do.

In addition, the controller 230 may transmit the corrected motion information to the display device 100.

Meanwhile, the method for controlling the display device according to the various embodiments described above may be implemented as a program and provided to a display device or an input device. In particular, a program including a method for controlling a display device may be stored and provided in a non-transitory computer readable medium.

The non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, and ROM.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

110: communication unit 120: image acquisition unit
130: control unit 140: image receiving unit
150: image processing unit 160: display unit
170: storage unit 180: audio output unit
190: input

Claims (20)

In the control method of the display device,
Receiving motion information of the portable terminal from a portable terminal, photographing the portable terminal to obtain a portable terminal image;
Converting the received motion information into a screen coordinate system aligned with a screen of the display device;
Correcting motion information converted to the screen coordinate system based on the mobile terminal image; And
Including; controlling the display device by using the corrected motion information,
The converting step,
Converting motion information measured based on a coordinate system of the mobile terminal into a global coordinate system aligned based on a geomagnetic direction; And
And converting the motion information converted to the global coordinate system into the screen coordinate system. The method of claim 1,
Motion information received from the mobile terminal,
The control method, which is a value measured based on a coordinate system of the mobile terminal aligned with respect to the mobile terminal. The method of claim 2,
The converting step,
When the camera coordinate system aligned with the camera for photographing the mobile terminal and the screen coordinate system are different, converting motion information converted into the screen coordinate system into the camera coordinate system. The method of claim 1,
The transformation parameter of the screen coordinate system is:
When an initialization input is input from a user while the portable terminal is aligned in a screen direction of the display device, the control method is calculated based on the shape of the portable terminal. The method of claim 1,
The step of correcting,
And compensating the motion information by comparing the size of the portable terminal obtained from the portable terminal image with the actual size of the previously obtained portable terminal. The method of claim 1,
The step of correcting,
Comparing the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system; And
When the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system do not match, the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system Synchronizing the; control method comprising a. The method of claim 1,
The controlling step,
Recognizing a three-dimensional motion of the mobile terminal using the corrected motion information; And
Controlling the display device based on the recognized three-dimensional motion. The method of claim 1,
The controlling step,
Detecting a motion pattern corresponding to each of the plurality of portable terminals by using the corrected motion information for each of the plurality of portable terminals when a plurality of portable terminals exist;
Recognizing the plurality of portable terminals using a motion pattern corresponding to each of the plurality of portable terminals. The method of claim 1,
The controlling step,
And controlling the display device based on the corrected motion information only when the portable terminal is photographed. The method of claim 1,
The controlling step,
Calculating pointing coordinates based on the corrected motion information; And
And displaying a pointer on the pointing coordinates. In the display device,
A communication unit for receiving motion information of the portable terminal from a portable terminal;
An image acquisition unit for capturing the mobile terminal to obtain a mobile terminal image; And
Converts the received motion information into a screen coordinate system aligned with the screen of the display device, corrects the motion information converted to the screen coordinate system based on the mobile terminal image, and uses the corrected motion information Includes; a control unit for controlling the display device,
The control unit,
A display device for converting motion information measured based on a coordinate system of the mobile terminal into a global coordinate system aligned based on a geomagnetic direction, and converting the motion information converted to the global coordinate system into the screen coordinate system. The method of claim 11,
Motion information received from the mobile terminal,
A display device that is a value measured based on a coordinate system of the portable terminal aligned with respect to the portable terminal. The method of claim 12,
The control unit,
When the camera coordinate system aligned with the camera for photographing the portable terminal and the screen coordinate system are different, the motion information converted to the screen coordinate system is converted into the camera coordinate system. The method of claim 11,
The transformation parameter of the screen coordinate system is:
When an initialization input is input by a user while the portable terminal is aligned in a screen direction of the display device, it is calculated based on a shape of the portable terminal. The method of claim 11,
The control unit,
And correcting the motion information by comparing the size of the portable terminal obtained from the portable terminal image with the actual size of the portable terminal. The method of claim 11,
The control unit,
The motion information of the portable terminal obtained from the portable terminal image is compared with the motion information converted to the screen coordinate system, and the motion information of the portable terminal obtained from the portable terminal image matches the motion information converted to the screen coordinate system. If not, synchronizing the motion information of the portable terminal obtained from the portable terminal image and the motion information converted to the screen coordinate system. The method of claim 11,
The control unit,
A display apparatus comprising: recognizing a three-dimensional motion of the mobile terminal using the corrected motion information, and controlling the display device based on the recognized three-dimensional motion. The method of claim 11,
The control unit,
When a plurality of portable terminals exist, a motion pattern corresponding to each of the plurality of portable terminals is sensed using the corrected motion information for each of the plurality of portable terminals, and a motion pattern corresponding to each of the plurality of portable terminals Display device, characterized in that to recognize the plurality of portable terminals by using. The method of claim 11,
The control unit,
And controlling the display device based on the corrected motion information only when the portable terminal is photographed. The method of claim 11,
It further includes a display unit,
The control unit,
And controlling the display unit to calculate pointing coordinates based on the corrected motion information and to display a pointer at the pointing coordinates.

Images