KR102516278B1 - User terminal for providing intuitive control environment on media pannel, server, and display apparatus - Google Patents

Abstract

Disclosed is a control method of a user terminal. The present control method comprises: a step of acquiring an image by photographing a display of a display device displaying a 3D image; a step of identifying, based on the acquired image, a gaze direction of the user terminal looking at the display; a step of detecting a motion of the user terminal to identify a motion direction of the user terminal; and a step of controlling the display device to display, on the display, a 3D image changed according to the motion of the user terminal by transmitting motion information comprising the gaze direction and a motion direction to the display device. Therefore, the present invention is capable of enhancing an interest and interaction of a user.

Description

User terminals, servers, and display devices that provide an intuitive control environment for media panels { USER TERMINAL FOR PROVIDING INTUITIVE CONTROL ENVIRONMENT ON MEDIA PANNEL, SERVER, AND DISPLAY APPARATUS }

The present disclosure relates to a system including a display device and a user terminal, and more particularly, to a system providing an intuitive user interface according to motion of a user terminal based on a positional relationship between the user terminal and the display device.

As there are many cases of displaying advertisements or exhibits through a display device having a conventional large screen, such a display structure not only creates an atmosphere of space, but also serves as a landmark of a specific place.

In particular, a method of elaborately expressing a three-dimensional object through a screen composed of a large flat display is in the spotlight. However, in this case, there may be a difference in sophistication depending on the viewing angle, so it is somewhat disappointing to fully satisfy the interest of users who look at the large display from various directions.

Registered Patent Publication No. 10-491432

The present disclosure provides a user terminal and system capable of interacting with a display device according to a gaze direction of a user looking at a display device including a large screen providing 3D images.

The objects of the present disclosure are not limited to the above-mentioned objects, and other objects and advantages of the present disclosure not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present disclosure. Further, it will be readily apparent that the objects and advantages of the present disclosure may be realized by means of the instrumentalities and combinations indicated in the claims.

A control method of a user terminal according to an embodiment of the present disclosure includes acquiring an image by photographing a display of a display device displaying a 3D image through a camera, by a processor, based on the acquired image, the display Identifying a gaze direction of the user terminal looking at, detecting motion of the user terminal through at least one motion sensor and identifying a motion direction of the user terminal, including the gaze direction and the motion direction and controlling the display device to display a 3D image changed according to the motion of the user terminal on the display by transmitting motion information to the display device through a communication unit.

In the identifying the gaze direction of the user terminal, based on the angle of the 3D image included in the image, the gaze direction of the user terminal based on the display may be identified.

In the identifying the gaze direction of the user terminal, the display included in the image may be identified, and the gaze direction of the user terminal based on the display may be identified based on the shape of the identified display.

The step of identifying the gaze direction of the user terminal may include identifying a location through a Global Positioning System (GPS) sensor, and when the user terminal is identified as existing within a predetermined distance from the display device according to the location, the obtaining Based on the image, it is also possible to identify the gaze direction of the user terminal relative to the display.

The step of identifying the direction of motion may include displaying an image captured by the camera in real time through a display of the user terminal, wherein the image displayed in real time includes at least a portion of the display of the display device. In this case, the method may include detecting motion of the user terminal through the motion sensor and identifying a direction of the motion.

In the controlling of the display device, the display device may be controlled so that at least one object in the 3D image displayed on the display moves according to the direction of motion based on the gaze direction of the user terminal. .

Here, the controlling of the display device may include displaying an image captured by the camera in real time through a display of the user terminal, at least within the display of the display device appearing in the center of the image displayed in real time. Identifying one coordinate, transmitting information on the identified coordinate to the display device so that at least one object in the 3D image displayed on the display of the display device moves according to the direction of motion around the coordinate It may also include controlling the display device so that the appearance is displayed.

A control method of a server according to an embodiment of the present disclosure includes controlling a display device to display a 3D image, receiving, from a user terminal, an image of a display of the display device displaying the 3D image captured, the Identifying a gaze direction of the user terminal looking at the display based on a received image, identifying a motion direction of the user terminal based on motion information of the user terminal received from the user terminal; and controlling the display device to display a 3D image changed according to the motion of the user terminal by transmitting motion information including the gaze direction and the motion direction to the display device.

The server control method includes controlling the display device to display a 3D image changed according to the motion of the first user terminal, continuously displaying the 3D image displayed on the display device a predetermined number of times according to the motion of the first user terminal. may include controlling the display device to display the changed 3D image according to the motion of the second user terminal.

According to an embodiment of the present disclosure, a control method of a display device including a display displaying a 3D image includes receiving, from a user terminal, an image captured by a display of the display device displaying the 3D image, the received Identifying a gaze direction of the user terminal looking at the display based on an image, identifying a motion direction of the user terminal based on motion information of the user terminal received from the user terminal, and displaying a 3D image changed according to the motion of the user terminal based on the direction and the motion direction.

A user terminal or display device according to the present disclosure can identify the gaze direction of the user terminal by utilizing only the photographing function of the user terminal in controlling a large electric signboard/screen or exhibition display, and intuitively according to the gaze direction and motion. It provides responsive 3D images to enhance user interest and interaction.

1 is a diagram for explaining operations of a user terminal and a display device performing interactions according to an embodiment of the present disclosure;
2 is a block diagram for explaining the configuration of a user terminal according to an embodiment of the present disclosure;
3 is a flowchart for explaining a control method of a user terminal according to an embodiment of the present disclosure;
4 is a diagram for explaining an operation of a user terminal for photographing and displaying a display device according to an embodiment of the present disclosure;
5 is a diagram for explaining an operation of changing and controlling a 3D image displayed on a display device based on motion recognition by a user terminal according to an embodiment of the present disclosure;
6 is a block diagram for explaining the configuration of a user terminal according to various embodiments of the present disclosure;
7 is a block diagram for explaining the configuration and operation of a display device according to an embodiment of the present disclosure, and
8 is a block diagram illustrating an operation performed by a server in association with one or more user terminals and one or more display devices according to an embodiment of the present disclosure.

Prior to a detailed description of the present disclosure, the method of describing the present specification and drawings will be described.

First, terms used in the present specification and claims are general terms in consideration of functions in various embodiments of the present disclosure. However, these terms may vary depending on the intention of a technician working in the art, legal or technical interpretation, and the emergence of new technologies. In addition, some terms are arbitrarily selected by the applicant. These terms may be interpreted as the meanings defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the art.

In addition, the same reference numerals or numerals in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of description and understanding, the same reference numerals or symbols are used in different embodiments. That is, even if all components having the same reference numerals are shown in a plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in the present specification and claims, terms including ordinal numbers such as “first” and “second” may be used to distinguish between elements. These ordinal numbers are used to distinguish the same or similar components from each other, and the meaning of the term should not be construed as being limited due to the use of these ordinal numbers. For example, the order of use or arrangement of elements associated with such ordinal numbers should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In the embodiments of the present disclosure, terms such as “module,” “unit,” and “part” are terms used to refer to components that perform at least one function or operation, and these components are hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except for cases where each of them needs to be implemented with separate specific hardware, so that at least one processor can be implemented as

Also, in an embodiment of the present disclosure, when a part is said to be connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that it may further include other components without excluding other components unless otherwise stated.

1 is a diagram for explaining operations of a user terminal and a display device performing interactions according to an embodiment of the present disclosure.

1 shows a user terminal 100 of a user and a display device 200 displaying a 3D image.

The user terminal 100 according to the present disclosure may correspond to various portable terminal devices such as a smart phone, a tablet PC, a notebook PC, a wearable device, a VR/AR device, and a PDA.

The display device 200 according to the present disclosure may correspond to various devices including one or more displays. For example, the display device 200 may include at least one screen, electronic display board, panel, etc. constituting a building or structure.

As a specific example, the display device 200 may include one or more flat or curved display panels, and the one or more display panels may display 3D images. Here, the 3D image may correspond to an object displayed on the surface of a flat or curved display, but expressed enough to be recognized as an object in a 3D space.

Referring to FIG. 1 , the user terminal 100 may capture at least a portion of the display of the display apparatus 200 by a user's manipulation. In this case, the user terminal 100 may identify the location and gaze direction of the user terminal 100 based on the display.

In this case, the user terminal 100 may directly or indirectly communicate with the display device 200 . Specifically, the user terminal 100 may interwork with the display device 200 based on identification information of the display device 200 captured. At this time, on the premise that the location of the user terminal 100 is within a certain distance from the location of the display device 200 or that the user terminal 100 photographs the display device 200, the user terminal 100 It may also work with the display device 200.

In addition, the user terminal 100 may acquire motion information according to the motion of the user terminal 100 according to the user's manipulation and transmit it to the display device 200 . Here, the motion information may include a position of the user terminal 100, a gaze direction, a motion direction, and the like.

In this case, based on the position and gaze direction of the user terminal 100, the display device 200 may display a 3D image changed according to the gaze direction and motion direction of the user terminal 100. For example, the display device 200 may reconstruct and display the 3D image so that at least one object in the 3D image moves or moves in the direction of the user terminal 100's eye line and motion.

As such, the display device 200 according to the present disclosure provides a plurality of users with 3D images viewed from various angles, providing an active 3D image that is changed according to the gaze direction of each user and the motion direction of each user's user terminal. can provide In addition, the user terminal 100 according to the present disclosure identifies the direction of the user's gaze only by photographing the display device 200, while the user's motion (the motion of the user terminal 100) is based on the direction of the user's gaze. ) has the advantage of maximizing the UX (User Experience) of each user by enabling the 3D image changed according to the display device 200 to be provided.

The configuration and operation of a user terminal or a display device will be described in more detail through the following drawings.

2 is a block diagram for explaining the configuration of a user terminal according to an embodiment of the present disclosure.

Referring to FIG. 2 , the user terminal 100 may include a camera 110, a communication unit 120, a motion sensor 130, a processor 140, and the like.

The camera 110 is a component for obtaining an image by photographing the surroundings of the user terminal 100 . The camera 110 may include at least one image sensor, and may be implemented as an RGB camera, a stereo camera, a Time of Flight (ToF) camera, or the like.

As an example, the camera 110 may acquire an image by photographing at least a portion of the display device 200 . When the display device 200 displays a 3D image, an image photographed by the camera 110 may include a 3D image displayed on the display of the display device 200 .

The communication unit 120 is a component for performing communication with various external devices, and may include circuits, modules, chips, etc. for performing communication with at least one external device through various wired/wireless communication methods.

The communication unit 120 may be connected to external devices through various networks.

The network may be a Personal Area Network (PAN), a Local Area Network (LAN), a Wide Area Network (WAN), etc., depending on the area or size, and an intranet, It may be an extranet or the Internet.

The communication unit 120 performs long-term evolution (LTE), LTE Advance (LTE-A), 5th generation (5G) mobile communication, code division multiple access (CDMA), wideband CDMA (WCDMA), and universal mobile telecommunications system (UMTS). , WiBro (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), Zigbee, etc. It can be connected with external devices through

In addition, the communication unit 120 may be connected to external devices through a wired communication method such as Ethernet, optical network, universal serial bus (USB), or thunderbolt.

The motion sensor 130 is a component for detecting motion of the user terminal 100 . The motion sensor 130 may include at least one of an acceleration sensor, a geomagnetic sensor, and a gyro sensor, and the user terminal 100 uses the motion sensor 130 to determine the posture of the user terminal 100 (ex. three mutually perpendicular positions). Rotation angle based on each axis of the dog), movement direction, movement speed, etc. can be identified in real time.

The processor 140 is a component for controlling the user terminal 100 as a whole. Specifically, the processor 120 may perform operations according to various embodiments of the present disclosure by executing at least one instruction stored in the memory 110 while being connected to the memory 110 .

The processor 140 may include a general-purpose processor such as a CPU, AP, or digital signal processor (DSP), a graphics-only processor such as a GPU or a vision processing unit (VPU), or an artificial intelligence-only processor such as an NPU. An artificial intelligence-only processor may be designed as a hardware structure specialized for training or use of a specific artificial intelligence model.

3 is a flowchart illustrating a control method of a user terminal according to an embodiment of the present disclosure.

Referring to FIG. 3 , the user terminal 100 may obtain an image by photographing the display of the display device 200 displaying a 3D image through the camera 110 (S310).

As an example, the user terminal 100 may activate a photographing function according to a user input. Here, the camera 110 may obtain an image by capturing the display device 200 according to a user's manipulation, and the corresponding image may be displayed on the display of the user terminal 100 .

At this time, the user terminal 100 may perform pairing with the display device 200 . This pairing may be performed by direct communication between the user terminal 100 and the display device 200 or by indirect communication through at least one external device (ex. relay device, server, etc.).

In this case, the user terminal 100 may extract the display device 200 within the image. Specifically, the user terminal 100 may identify the display of the display device 200 by utilizing at least one artificial intelligence model (ex. CNN) trained to identify at least one display/screen in an image. Here, the user terminal 100 may obtain identification information of the display device 200 based on at least one of the shape and size of the display of the display device 200 and the surroundings of the display device 200 . The identification information may include information about an identification number, location, and the like of the display device 200 . To this end, information on the shape, size, and surroundings of the display of one or more display devices may be pre-stored in the memory of the user terminal 100 to match the identification information of each display device.

Alternatively, as it is identified that at least one display device is included in the image, the user terminal 100 may determine the location of the user terminal 100 . At this time, the user terminal 100 identifies the location of the user terminal 100 through a GPS (Global Positioning System) sensor, or the location of at least one relay device (ex. Wi-Fi repeater) connected to the user terminal 100. Based on this, the location of the user terminal 100 can be identified. Here, the user terminal 100 selects location information close to the current location of the user terminal 100 from among the pre-stored location information of one or more display devices, and identifies the display device (ex. 100) matching the selected location information. Pairing can be performed based on

And, referring to FIG. 3 , based on the image obtained through the camera 110, the user terminal 100 may identify the gaze direction of the user terminal relative to the display of the display device 200 (S320). ). At this time, the user terminal 100 may identify the gaze direction on the premise that the user terminal 100 exists within a certain distance from the display device 200 . Here, the location of the user terminal 100 identified through the GPS sensor may be compared with the location of the display device 200 .

In one embodiment, the user terminal 100 based on the angle of the 3D image (: 3D image displayed on the display of the display device 200) included in the image, the user terminal (based on the display of the display device 200) ( 100) can be identified.

In this case, first, the user terminal 100 may extract a 3D image included in the image. Here, at least one artificial intelligence model (ex. CNN) trained to identify 3D images or moving images may be utilized. Alternatively, the 3D image displayed on the display may be identified as the display is extracted through at least one artificial intelligence model trained to identify the display/screen within the image.

Specifically, the user terminal 100 identifies the angle of the 3D image in the image by comparing the image data for the 3D image displayed on the display device 200 with the 3D image in the acquired image, and the user terminal ( 100) can determine the gaze direction. Here, the image data for the 3D image may be previously stored in the user terminal 100, may be received from the paired display device 200, or may be received from the server 300 described later.

Alternatively, as an embodiment, the user terminal 100 may identify the gaze direction of the user terminal 100 based on the shape of the display/screen of the display device 200 included in the image.

Specifically, the user terminal 100 may extract the display/screen of the display device 200 within the image by utilizing at least one artificial intelligence model trained to identify the display/screen within the image.

Here, the user terminal 100 may identify the gaze direction (of the camera) of the user terminal 100 looking at the display/screen of the display device 200 based on the extracted shape of the display/screen. For example, the shape of the display/screen of the display device 200 may be pre-stored for each of various gaze directions toward the display device 200 . In this case, the user terminal 100 compares the shape of the display/screen in the image with the shape of the display/screen previously stored for each gaze direction to determine the gaze direction of the user terminal 100 looking at the display/screen in the image. can

Meanwhile, unlike S320 of FIG. 3 , the user terminal 100 may determine the gaze direction of the user terminal 100 based on the location of the user terminal 100 acquired based on the GPS sensor. Specifically, the user terminal 100 is the user terminal 100 looking at the display device 200 based on the relationship between the current location of the user terminal 100 identified through the GPS sensor and the position of the display device 200. direction of gaze can be identified.

When the gaze direction is identified according to at least one of the various embodiments described above, the user terminal 100 detects the motion of the user terminal 100 through the motion sensor 130 to identify the motion direction of the user terminal 100. It can (S330).

At this time, motion detection of the user terminal 100 may be performed on the assumption that at least a part of the display apparatus 200 is being photographed by the user terminal 100 .

In this regard, as an example, an image captured through the camera 110 may be displayed in real time through the display of the user terminal 100 . Here, when the image displayed in real time includes at least a part of the display of the display device 200 (: when at least a part of the display of the display device 200 or a 3D image is extracted from the image), the user terminal 100 ) may detect the motion of the user terminal 100 through the motion sensor 130 and identify the direction of the motion. That is, motion detection of the user terminal 100 may be activated only while the display/screen of the display device 200 is photographed through the camera 110 of the user terminal 100 .

Motions include, for example, pushing the user terminal 100 in a direction closer to the display device 200, pulling the user terminal 100 in a direction away from the display device 200, moving up/down/left/right, rotating, up and down. / It may include, but is not limited to, shaking left and right.

And, referring to FIG. 3 , the user terminal 100 transmits motion information including the above-described gaze direction and motion direction to the display device 200 to display a 3D image changed according to the motion of the user terminal 100. The display device 200 may be controlled to do so (S340).

For example, a state in which at least one object in the 3D image displayed on the display of the display device 200 moves according to the direction of motion based on the gaze direction of the user terminal 100 may be expressed.

Specifically, the user terminal 100 may display an image captured by the camera 110 in real time, and identify at least one coordinate in the display of the display device 200 appearing in the center of the image displayed in real time. can

In this case, the user terminal 100 may transmit information about the identified coordinates to the display device 200 . As a result, a state in which at least one object in the 3D image displayed on the display of the display device 200 moves along the direction of motion around the above-described coordinates can be expressed.

4 to 5 are drawings for explaining an operation of a user terminal for photographing and displaying a display device and an operation of a display device for displaying a 3D image changed according to a motion of the user terminal according to an embodiment of the present disclosure.

Referring to FIG. 4 , the user terminal 100 may display an image 410 captured through the camera 110 . In this case, the corresponding image 410 may include at least a part of the display 210 of the display device 200 .

Here, the user terminal 100 may identify at least one coordinate within the display 210 corresponding to the center of the image 410 . For example, the user terminal 100 may select at least one coordinate in the display 210 that matches the center line 411 . And, the user terminal 100 may transmit information about the selected coordinates to the display device 200 .

In a state in which the user terminal 100 displays the above-described image 410 as shown in FIG. 4 , the user pushes the user terminal 100 in a direction closer to the display device 200 or pulls it in the opposite direction as shown in FIG. 5 . of motion can be performed.

For example, when the user terminal 100 is pushed in a direction closer to the display device 200, the motion of the user terminal 100 may be detected, and the direction of the motion (: the direction closer to the display device 200) ) and motion information including the gaze direction of the user terminal 100 may be transmitted to the display device 200 .

As a result, as shown in FIG. 5 , objects in the 3D image displayed on the display device 200 (ex. water droplets, particles corresponding to water) may move according to the direction of motion around the user's gaze direction. . Referring to FIG. 5 , since the motion of the user terminal 100 being pushed along the gaze direction is detected, a 3D image in which water particles are pushed left and right like Moses' miracle centered on the gaze direction (motion direction) is displayed on the display device 200. ) can be expressed on At this time, water droplets may be pushed left and right with coordinates on the display 210 that match the center line 411 in the image 410 captured through the user terminal 100 as the center.

As a result, the user can feel as if he or she has pushed objects (eg, particles corresponding to water) in the 3D image, and interest and interest according to intuitive interaction can be increased.

Meanwhile, as an embodiment, after the motion information obtained through the user terminal 100 is transmitted to the display device 200, the user terminal 100 may monitor a 3D image displayed through the display device 200.

Specifically, even after the motion information is transmitted, the user terminal 100 may sequentially acquire a plurality of images by continuously photographing the display/screen of the display device 200 through the camera 110 . At this time, the user terminal 100 may identify whether or not the 3D image is changed, the degree of change, and the changed shape/shape by analyzing the plurality of images acquired sequentially.

For example, when a change in a 3D image is not identified through images sequentially captured for a certain period of time even though motion information is transmitted, the user terminal 100 sends the display device 200 and/or the server 300 to be described later. A request for checking the communication state or the state of the display device 200 may be transmitted. At this time, the user terminal 100 may output a message informing that there is an abnormality in the current communication state or the state of the display device 200 .

For example, if the change of the 3D image is identified after the motion information is transmitted, but the direction/speed of the motion matching the motion information does not match the change speed of the actually changed 3D image, the user terminal 100 also displays A request for checking the communication state or the state of the display device 200 may be transmitted to the device 200 and/or the server 300 to be described later.

In this regard, as an example, when the display/screen of the display device 200 is not included in the image captured through the camera 110 after the motion information is transmitted, the user terminal 100 displays the A request message requesting that the direction (the direction of the camera 110) be maintained toward the display device 200 may be output. In this case, the above-described monitoring of the display device 200 performed after motion information is transmitted can be smoothly performed.

However, it is identified that the change of the 3D image monitored after the motion information is transmitted is matched without problems for continuous motions (e.g., one motion corresponding to a standard unit) of a certain number of times (eg, twice) or more. (: A change in the 3D image that matches the motion information appears), the user terminal 100 may not monitor the 3D image after the motion information is transmitted for the subsequent motion. That is, in this case, continuous shooting (camera 110) may not be performed for a certain time immediately after the motion information is transmitted (the time when the 3D image is changed). As a result, unnecessary energy consumption and load consumption in the post-monitoring process can be minimized.

6 is a block diagram for explaining the configuration of a user terminal according to various embodiments of the present disclosure.

Referring to FIG. 6 , the user terminal 100 may further include a user input unit 150, a display 160, and the like in addition to a camera 110, a communication unit 120, a motion sensor 130, and a processor 140. .

The user input unit 150 is a component for receiving various commands or information from a user. The user input unit 150 may be implemented with at least one button, a touch pad, a touch screen, a microphone, a camera, a sensor, and the like. In addition, the user terminal 100 may be connected to a separate user input device composed of a mouse, keyboard, or the like.

For example, based on the user input received through the user input unit 150, at least one instruction or program for executing the entire process of FIG. 3 described above may be driven. Also, pairing between the user terminal 100 and the display device 200 may be initiated based on a user input received through the user input unit 150 .

The display 160 may be implemented with a Liquid Crystal Display (LCD), Plasma Display Panel (PDP), Organic Light Emitting Diodes (OLED), Transparent OLED (TOLED), Micro LED, etc., but is not limited thereto, and is not limited thereto. Various known types of displays may be included. The display 160 may be implemented in the form of a touch screen capable of detecting a user's touch manipulation, or may be implemented as a flexible display capable of being folded or bent.

As an example, the user terminal 100 may display an image captured by the camera 110 (eg, including at least a portion of the display device 200 displaying a 3D image) on the display 160 .

Meanwhile, FIG. 7 is a block diagram for explaining the configuration and operation of a display device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 7 , the display device 200 may include a display 210, a communication unit 220, a processor 230, and the like.

The display 210 is a component for displaying various images. As an example, the display 210 may include a plurality of display panels for constituting a large electric signboard or a large screen, and may display the above-described 3D image. In this case, the 3D image may be expressed in the form of a video according to a plurality of image frames provided sequentially.

Through the communication unit 220, the display device 200 may communicate with one or more user terminals and servers.

The processor 230 is a component for controlling the above-described components (the display 210 and the communication unit 220) of the display device 200.

On the other hand, although the above-described embodiments have been mainly described for the embodiment in which the user terminal 100 identifies the gaze direction of the user terminal 100, the process of identifying the gaze direction of the user terminal 100 is a display device ( 200) may be performed.

As an example, the display device 200 including the display 210 displaying a 3D image may receive, from the user terminal 100, an image captured by the display of the display device 200 on which the 3D image is displayed. .

Based on the received image, the display device 200 may identify the gaze direction of the user terminal 100 looking at the display 210 . In this case, the display device 200 utilizes at least one of the shape of the display 210 in the received image, the angle of the 3D image included in the received image, and the position of the user terminal 100 to determine the position of the user terminal 100. direction of gaze can be determined.

Also, based on the motion information of the user terminal 100 received from the user terminal 100, the display device 200 may identify the motion direction of the user terminal. In this case, the motion information may include information about the direction of motion of the user terminal 100, the speed of motion, and the posture of the user terminal 100.

Here, based on the gaze direction and motion direction of the user terminal 100, the display device 200 reconstructs (changes) the 3D image according to the motion of the user terminal 100, and displays the reconstructed 3D image through the display 210. images can be displayed.

Meanwhile, FIG. 8 is a block diagram illustrating an operation performed by a server in association with one or more user terminals and one or more display devices according to an embodiment of the present disclosure.

The server 100 is a configuration for performing communication with one or more user terminals and one or more display devices, may include one or more computers, and may be provided in the form of a system.

The server 100 may interwork with one or more user terminals through at least one web page, application, or the like, and may support interworking between the user terminal and the display device.

Meanwhile, the above-described process of identifying the gaze direction of the user terminal 100 may be performed through the server 100 .

As an example, the server 100 may control at least one display device 200 to display a 3D image. Here, the 3D image may be pre-stored in the display device 200 or pre-stored on the server 100 .

At this time, the server 100 may receive, from the user terminal 100 , an image captured by the display 210 of the display device 200 on which the 3D image is displayed.

In this case, based on the received image, the server 300 may identify the gaze direction of the user terminal 100 looking at the display 210 . Similarly, the gaze direction of the user terminal 100 may be calculated through at least one of the shape of the display 210 in the received image, the angle of the 3D image included in the received image, and the position of the user terminal 100 .

Also, based on the motion information of the user terminal 100 received from the user terminal 100, the server 300 may identify a motion direction of the user terminal 300. The motion information may include information about a direction of motion of the user terminal 100, a speed of the motion, and a posture of the user terminal 100.

Here, the server 300 transmits motion information including a gaze direction and a motion direction to the display device 200 to control the display device 200 to display a 3D image changed according to the motion of the user terminal 100. can Alternatively, the server 300 may generate a 3D image changed according to the motion of the user terminal 100 by itself and then transmit the generated 3D image to the display device 200 .

Meanwhile, referring to FIG. 8 , the server 300 may communicate with a plurality of user terminals 100-1, 2, ..., n and a plurality of display devices 200-1, 2, ..., m. there is. Here, the server 300 may provide motion-based 3D images by linking one or more user terminals to each display device.

In this case, the 3D image may be changed as a result of simultaneously reflecting the motions of a plurality of user terminals on one display device, or the 3D image may be changed as a result of reflecting only the motion of one user terminal on one display device.

In this case, the server 300 may temporally separate a process of motion control of a 3D image by each user terminal with respect to one display device.

As an embodiment, the server 300 sets the priority of user terminals based on at least one of the order of proximity within a certain distance from the display device 200-1 and the order of pairing with the display device 200-1. can Alternatively, the server 300 may execute applications provided by the server 300 to prioritize user terminals according to the order in which motion control functions for the display device 200-1 are activated.

For example, it is assumed that the user terminal 100-1 approaches the display device 200-1 faster than the user terminal 100-2 and activates a motion control function.

In this case, the server 300 first sets the user terminal 100-1 as a main control device, and sets the display device 200-1 to display a 3D image changed according to the motion of the user terminal 100-1. You can control it. At this time, the user terminal 100-1 may provide a message informing that the 3D image may be changed according to the current motion of the user terminal 100-1. On the other hand, the user terminal 100 - 2 may inform that waiting is necessary because the 3D image is currently being changed according to the motion of another user terminal, and may also provide an expected waiting time.

However, when the 3D image displayed on the display device 200-1 is continuously changed a certain number of times according to the motion of the user terminal 100-1 (ex. when the motion of the reference unit is sequentially detected more than a certain number of times), The server 300 may set the next step, the user terminal 100-2, as the main control device.

In this case, the server 300 may control the display device 200-1 to display a 3D image changed according to the motion of the user terminal 100-2.

Meanwhile, the various embodiments described above may be implemented by combining two or more embodiments as long as they do not conflict or contradict each other.

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

According to the hardware implementation, the embodiments described in this disclosure are application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). ), processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

In some cases, the embodiments described herein may be implemented by a processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules described above may perform one or more functions and operations described herein.

On the other hand, computer instructions or computer programs for performing processing operations in the user terminal 100, the display device 200, or the server 300 according to various embodiments of the present disclosure described above are non-temporary computer It may be stored in a non-transitory computer-readable medium. A computer instruction or computer program stored in such a non-transitory computer readable medium is executed by a processor of a specific device in the user terminal 100, the display device 200, or the server 300 according to various embodiments described above. The processing operation is performed by the specific device described above.

A non-transitory computer readable medium is a medium that stores data semi-permanently and is readable by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of the non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is common in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: user terminal 200: display device
300: server

Claims (10)

In the control method of the user terminal,
Obtaining an image by photographing a display of a display device displaying a 3D image through a camera;
identifying, by a processor, a gaze direction of the user terminal looking at the display, based on the acquired image;
detecting a motion of the user terminal through at least one motion sensor and identifying a motion direction of the user terminal; and
Controlling the display device to display a 3D image changed according to the motion of the user terminal on the display by transmitting motion information including the gaze direction and the motion direction to the display device through a communication unit; and
The step of controlling the display device is,
Controlling the display device so that at least one object in the 3D image displayed on the display is displayed moving according to the direction of the motion based on the direction of the user terminal's gaze;
The step of controlling the display device is,
displaying an image taken by the camera in real time through a display of the user terminal;
identifying at least one coordinate in the display of the display device appearing at the center of the image displayed in real time; and
Information on the identified coordinates is transmitted to the display device so that at least one object in the 3D image displayed on the display of the display device moves according to the direction of motion around the coordinates, the display device A control method of a user terminal, including; controlling a. According to claim 1,
Identifying the gaze direction of the user terminal,
Based on the angle of the 3D image included in the image, the user terminal control method of identifying a gaze direction of the user terminal with respect to the display. According to claim 1,
Identifying the gaze direction of the user terminal,
identify the display included in the image;
A method of controlling a user terminal, wherein a gaze direction of the user terminal based on the display is identified based on the shape of the identified display. According to claim 1,
Identifying the gaze direction of the user terminal,
Identify location through Global Positioning System (GPS) sensor,
When the user terminal is identified as existing within a certain distance from the display device according to the location, based on the acquired image, the control of the user terminal to identify the gaze direction of the user terminal with respect to the display. method. According to claim 1,
Identifying the direction of the motion,
displaying an image taken by the camera in real time through a display of the user terminal; and
When the image displayed in real time includes at least a portion of the display of the display device, detecting motion of the user terminal through the motion sensor and identifying a direction of the motion; controlling the user terminal, including method. delete delete In the server control method,
controlling a display device to display a 3D image;
Receiving, from a user terminal, an image of the display of the display device on which the 3D image is displayed;
Identifying a gaze direction of the user terminal looking at the display based on the received image;
identifying a motion direction of the user terminal based on motion information of the user terminal received from the user terminal; and
Controlling the display device to display a 3D image changed according to the motion of the user terminal by transmitting motion information including the gaze direction and the motion direction to the display device;
The step of controlling the display device is,
Controlling the display device so that at least one object in the 3D image displayed on the display is displayed moving according to the direction of the motion based on the direction of the user terminal's gaze;
The step of controlling the display device is,
displaying the captured image in real time through a display of the user terminal;
identifying at least one coordinate in the display of the display device appearing at the center of the image displayed in real time; and
Information on the identified coordinates is transmitted to the display device so that at least one object in the 3D image displayed on the display of the display device moves according to the direction of motion around the coordinates, the display device Controlling the; including, the control method of the server. According to claim 8,
The control method of the server,
controlling the display device to display a 3D image changed according to the motion of the first user terminal; and
Controlling the display device to display the changed 3D image according to the motion of the second user terminal when the 3D image displayed on the display device is continuously changed by a predetermined number of times according to the motion of the first user terminal. , how to control the server. In the control method of a display device including a display displaying a 3D image,
Receiving, from a user terminal, an image of the display of the display device on which the 3D image is displayed;
Identifying a gaze direction of the user terminal looking at the display based on the received image;
identifying a motion direction of the user terminal based on motion information of the user terminal received from the user terminal; and
Based on the gaze direction and the motion direction, displaying a 3D image changed according to the motion of the user terminal; and
Displaying the 3D image
At least one object in the 3D image is displayed so that a moving state is expressed according to the direction of motion based on the gaze direction of the user terminal,
The step of displaying the 3D image,
As the captured image is displayed in real time through the display of the user terminal, when the user terminal identifies at least one coordinate in the display of the display device appearing at the center of the image displayed in real time,
Of the display device for receiving information on the identified coordinates from the user terminal and displaying a state in which at least one object in the 3D image displayed on the display of the display device moves according to the direction of motion around the coordinates control method.

Images