KR102294919B1 - Terminal apparatus for displaying hologram image corresponding to the user's gaze and hand movement event and control method thereof - Google Patents

Abstract

The present invention relates to a terminal device for outputting a holographic image and a method for controlling the same, comprising: a hologram device coupling unit formed in the terminal device to electrically couple at least one hologram device; and generating hologram data to be output through the hologram device when a predetermined specific event occurs in the terminal device in connection with the operation of the terminal device while controlling the operation of the terminal device, or generating previously generated hologram data and a control unit for calling and delivering to the hologram device.

Description

A terminal device that outputs a holographic image corresponding to a user's gaze and hand movement event and a control method thereof

The present invention relates to a terminal device for outputting a holographic image corresponding to a user's gaze and hand motion event and a control method thereof, and more particularly, to a hologram device coupled to the terminal device when a preset event occurs in the terminal device. The present invention relates to a terminal device for outputting a holographic image corresponding to a user's gaze and hand movement event, which enables the output of a three-dimensional stereoscopic hologram image through a terminal device and a method for controlling the same.

In general, a holographic image is a three-dimensional picture that looks three-dimensional just like the real thing, and a hologram is made using the principle of holography.

Generally, a hologram that can be easily viewed includes a hologram attached to a credit card, and a reflective hologram is attached to the credit card to prevent counterfeiting.

As such, in the past, for counterfeit prevention or authentication, a method of recording a unique pattern on a holographic photosensitive material (eg, a film) using a laser and re-projecting the laser onto the holographic photosensitive material was mainly used.

However, recently, a three-dimensional space created using a reflective body (or a translucent body or floating glass) of a specific shape (eg, inverted pyramid shape, cylindrical shape, etc.) ), a holographic device that can

However, these hologram devices (or hologram output devices) are not standardized in the format of input/output data, communication format, size of hologram output image, and resolution of hologram output image. A holographic device is being released.

Background art of the present invention is disclosed in Korean Patent Laid-Open No. 10-2017-0092106 (published on Aug. 10, 2017, image display device).

According to one aspect of the present invention, the present invention was created to solve the above problems, and when a preset event occurs in the terminal device, it is possible to output a 3D stereoscopic hologram image through a hologram device coupled to the terminal device. An object of the present invention is to provide a terminal device for outputting a holographic image corresponding to an event of a user's gaze and hand movement, and a method for controlling the same.

A terminal device for outputting a holographic image according to an aspect of the present invention includes: a hologram device coupling unit formed in the terminal device to electrically couple at least one hologram device; and generating hologram data to be output through the hologram device when a predetermined specific event occurs in the terminal device in connection with the operation of the terminal device while controlling the operation of the terminal device, or generating previously generated hologram data It characterized in that it includes; a control unit for calling and delivering to the hologram device.

In the present invention, the hologram device type detection unit for detecting the type information of the hologram device coupled to the hologram device coupling unit; and a hologram adapter unit for converting the data format according to the type of the hologram device and transmitting the hologram data to the hologram device.

In the present invention, the control unit is characterized in that it is implemented so that a function of converting hologram data, which is a function of the hologram adapter unit, according to the type of the hologram device, can be built-in without separately forming a hologram adapter unit.

In the present invention, the hologram device coupling unit is a position where a user can easily view an image or a moving picture output from the hologram device even while operating the terminal device, and at least one of a button, a lever, and a jog dial is installed. It is characterized in that it is formed on one side of at least one of the left, right, and center of the control panel.

In the present invention, the hologram device type detection unit is characterized in that it detects type information of at least one of a model name, a manufacturer, a power supply standard, a communication standard, a resolution of an image or video to be reproduced, and an audio standard of the hologram device.

In the present invention, the hologram device type detection unit is implemented to directly receive the type information of the hologram device through a communication method designated by the hologram device, or to receive published catalog information in a manual method. do.

In the present invention, further comprising; a gaze tracking unit for detecting and tracking the user's gaze or face direction, wherein the control unit is a hologram of a position corresponding to the gaze or face direction of the user when a specified event occurs in the terminal device It is characterized in that the holographic image is outputted in a direction corresponding to the user's gaze through the device.

In the present invention, the control unit detects and tracks the user's gaze or face direction during operation of the terminal device, and when the user's gaze or face direction faces downward of a preset first reference area, the holographic image of the user Outputs from the lower part to the upper part in response to the gaze, and when the user's gaze or face direction is directed toward the preset first reference area, the holographic image is simultaneously displayed from the upper part and the lower part toward the first reference area in response to the user's gaze. It is characterized in that the hologram image is outputted from the top to the bottom in response to the user's gaze when the user's gaze or face direction is directed toward the top of the preset first reference area.

In the present invention, when the user's gaze or face direction is higher than the terminal device, the controller outputs the holographic image through the combined hologram device so as to output the holographic image in the air higher than the terminal device. characterized in that

In the present invention, a hand tracking unit for detecting and tracking the direction and speed of the user's hand movement is included, wherein the control unit detects and tracks the direction and speed of the user's hand movement while the holographic image is already output through the hologram device The speed is detected and tracked, and as a result of detecting and tracking the movement of the user's hand, when the hand moves toward the output holographic image, it corresponds to the direction and speed of the hand movement in the already output holographic image. It is characterized in that by applying a predetermined effect, the hologram data to which the specified effect is applied is output in response to the user's gaze or face direction.

According to another aspect of the present invention, there is provided a method of controlling a terminal device for outputting a holographic image, the method comprising: checking, by a controller of a terminal device for outputting a holographic image, whether the hologram device is coupled to the terminal device; when the hologram device is coupled to the terminal device, checking, by the controller, whether a predetermined specific event for outputting hologram data from the terminal device occurs; and when a predetermined specific event for outputting hologram data occurs, the controller transmits the designated hologram data to the hologram device so that the hologram device outputs the hologram data.

In the present invention, when the hologram device is coupled to the terminal device, the method comprising: detecting, by the controller, type information of the hologram device through a hologram device type detection unit; converting, by the controller, a format of the hologram data to be output according to the event to a type of the hologram device when a predetermined specific event for outputting the hologram data occurs; and transmitting the hologram data converted according to the type of the hologram device to the hologram device.

In the present invention, the conversion of the format of the hologram data according to the type of the hologram device may be performed directly by the controller or through a separate hologram adapter.

In the present invention, in the step of detecting the type information of the hologram device, the hologram device type detection unit includes at least one of a model name, a manufacturer, a power supply standard, a communication standard, a resolution of an image or moving image to be reproduced, and an audio standard of the hologram device. It is characterized in that one or more types of information are detected.

In the present invention, the hologram device type detection unit is implemented to directly receive the type information of the hologram device through a communication method designated by the hologram device, or to receive published catalog information in a manual method. do.

In the present invention, the control step of detecting and tracking the user's gaze or face direction through the gaze tracking unit; and when a specified event occurs in the terminal device, outputting, by the controller, a holographic image toward a direction corresponding to the user's gaze through the hologram device at a position corresponding to the user's gaze or face direction. characterized.

In the present invention, in the step of outputting the holographic image toward the direction corresponding to the user's gaze, the control unit transmits the holographic image to the user when the user's gaze or face direction is below a preset first reference area. Outputs from the lower part to the upper part corresponding to the gaze of the user, and when the user's gaze or face direction is directed toward the preset first reference area, the holographic image is simultaneously displayed from the upper part and the lower part toward the first reference area in response to the user's gaze Alternatively, the hologram image is outputted from the top to the bottom in response to the user's gaze when the user's gaze or face direction is directed toward the top of the preset first reference area.

In the present invention, in the step of outputting the holographic image toward the direction corresponding to the user's gaze, the control unit, when the user's gaze or face direction is higher than the terminal device, the high air space than the terminal device It is characterized in that the holographic image is output through a hologram device coupled to output the holographic image.

In the present invention, the step of the controller detects and tracks the direction and speed of the movement of the user's hand through the hand tracking unit; and as a result of detecting and tracking the movement of the user's hand, when the hand moves toward the output holographic image, the control unit is preset in response to the direction and speed of the hand movement in the holographic image that is already being output. and outputting the hologram data to which the specified effect is applied in response to the user's gaze or face direction by applying an effect.

According to one aspect of the present invention, when a preset event occurs in the terminal device, the 3D stereoscopic hologram image is output through the hologram device coupled to the terminal device, thereby satisfying the user's curiosity and increasing the fun. In addition, there is an effect of increasing the satisfaction of the terminal device user.

1 is an exemplary diagram illustrating a schematic configuration of a terminal device for outputting a holographic image according to a first embodiment of the present invention;
2 is an exemplary diagram illustrating a schematic configuration of a terminal device for outputting a holographic image according to a second embodiment of the present invention;
3 is an exemplary diagram for explaining a schematic shape and a control method of a terminal device for outputting a holographic image according to an embodiment of the present invention;
4 is a flowchart illustrating a method of controlling a terminal device for outputting a holographic image according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a schematic configuration of a terminal device for outputting a holographic image according to a third embodiment of the present invention.
6 is an exemplary view for explaining a method of outputting a hologram image in various directions according to a user's gaze in FIG. 5 .
7 is a flowchart for explaining a method of outputting a hologram image in various directions according to a user's gaze in FIG. 5;
8 is an exemplary diagram illustrating a schematic configuration of a terminal device for outputting a holographic image according to a fourth embodiment of the present invention.
FIG. 9 is an exemplary view for explaining a method of outputting hologram data by changing according to the movement of a user's hand in FIG. 8;
FIG. 10 is a flowchart for explaining a method of changing and outputting hologram data according to the movement of a user's hand in FIG. 8;

Hereinafter, an embodiment of a terminal device for outputting a holographic image and a method for controlling the same according to the present invention will be described with reference to the accompanying drawings.

In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

1 is an exemplary diagram illustrating a schematic configuration of a terminal device for outputting a holographic image according to a first embodiment of the present invention.

As shown in FIG. 1 , the terminal device 200 for outputting a holographic image according to the first embodiment of the present invention includes a hologram device coupling unit 210 , a hologram device type detection unit 220 , a control unit 230 , and a hologram adapter unit 240 .

The hologram device coupling unit 210 is formed in the terminal device 200 to physically and electrically couple at least one hologram device 100 (eg, wired or wireless).

For example, the hologram device coupling unit 210 is formed at a position where a user can easily view an image (or a moving picture) output from the hologram device 100 while operating the terminal device. For example, as shown in FIG. 3 , it may be formed on one side (eg, left, center, right, etc.) of a terminal device manipulation unit (eg, a panel on which at least one of a button, a lever, and a jog dial is installed). Alternatively, as shown in FIG. 6 , it may be formed on the monitor (or display screen), and the position of the hologram device 100 may be changed according to a direction in which the hologram is output (or displayed).

The hologram device coupling unit 210 supplies power to the coupled hologram device 100 and at the same time, data to be output through the hologram device 100 (ie, information, image, or video data transmitted from a terminal device, e.g. : Transmits event video, information video, guide video, avatar video, broadcast video, game video, CCTV video, etc.) through a predetermined communication method.

Accordingly, the hologram device 100 receives the received data (ie, information, images, or video data transmitted from the terminal device, such as event images, information images, guide images, avatar images, broadcast images, game images, CCTV images, etc.) are output as a three-dimensional hologram. At this time, the output data (that is, information, video, or video data transmitted from the terminal device, e.g., event video, information video, guide video, avatar video, broadcast video, game video, CCTV video, etc.) It is different from the method of outputting a unique pattern using a laser (actually, the laser reflected on the pattern is displayed).

When the hologram device 100 is coupled to the hologram device coupling unit 210 as described above, the hologram device type detection unit 220 detects the type of the coupled hologram device 100 .

For example, the hologram device type detection unit 220 detects type information such as a model name, a manufacturer, a power supply standard, a communication standard, a resolution of an image (movie) to be reproduced, and an audio standard of the hologram device 100 .

The type information of the hologram device may be directly provided by the hologram device 100 through a designated communication method, or may be input through a user by referring to published catalog information.

When the type information of the hologram device 100 detected as described above is detected, the hologram device type detection unit 220 transmits the type information to the control unit 230 .

The controller 230 controls the operation of the terminal device 200 .

In addition, when a preset event (eg, program execution completion, information reception, information output, etc.) occurs in the terminal device 200 in connection with the operation of the terminal device, the controller 230 sends the terminal device 200 to the terminal device 200 . The combined hologram device 100 generates and transmits hologram data to be output (eg, 3D stereoscopic hologram image data).

In this case, hologram data corresponding to the event situation may be generated in advance and stored in an internal memory (not shown). Therefore, generating and transmitting the hologram data is a concept including retrieving (or loading) the hologram data stored in the memory (not shown) and transferring the hologram data.

Accordingly, when the event situation occurs, the controller 230 transmits the hologram data previously generated in response to the event situation to the hologram device 100 to be output.

For example, when the event (eg, program execution completion, information reception, information output, etc.) occurs, a human-shaped avatar (eg, male or female) appears on the hologram device 100 and applauds to guide the event situation. It can alert information to be used, explain how to use the terminal device 200 as if it were a secretary, or explain precautions or guidance according to an event situation, or additional tasks to be performed.

Meanwhile, when the controller 230 outputs the hologram data corresponding to the event situation as described above, the hologram adapter 240 changes the data format so that the hologram data is optimized for the type of the hologram device 100 .

For example, a data format is changed (converted) according to type information such as a model name, manufacturer, power supply standard, communication standard, resolution of an image (movie) to be reproduced, and audio standard of the hologram device 100 .

When the data format is changed (converted) as described above, the hologram adapter unit 240 connects (or combines) the changed (converted) hologram data to the hologram device coupling unit 210 (or the corresponding hologram device 100). forward to

Accordingly, the hologram device 100, which has received the hologram data changed (converted) into a data format optimized for its type information, outputs the hologram data, thereby satisfying the user's curiosity and increasing the fun of using the terminal device. be able to do

2 is an exemplary diagram illustrating a schematic configuration of a terminal device for outputting a holographic image according to a second embodiment of the present invention.

In the embodiment shown in FIG. 1, the hologram adapter unit 240 is separately formed, and the hologram data output from the control unit 230 is converted to be optimized for the type of the combined hologram device 100 and output. was explained about.

However, without separately forming the hologram adapter unit 240 , as shown in FIG. 2 , the function of the hologram adapter unit 240 (that is, hologram data is optimized for the type of the combined hologram device 100 ). conversion function) may be built into the control unit 230 .

As shown in FIG. 2 , when the function of the hologram adapter unit 240 (ie, a function of converting hologram data to be optimized for the type of the combined hologram device 100 ) is incorporated in the control unit 230 . , when the control unit 230 generates hologram data corresponding to an event, immediately converts it into hologram data optimized for the type of the hologram device 100 and outputs the hologram data, thereby outputting the hologram data to the hologram device coupling unit 210 The connected (or coupled) hologram device 100 is directly transmitted and output. Accordingly, it is possible to simply implement the configuration of the terminal device.

3 is an exemplary diagram for explaining a schematic shape and a control method of a terminal device for outputting a holographic image according to an embodiment of the present invention.

As shown in FIG. 3 , the hologram device 100 is specific in a three-dimensional space (eg, a transparent capsule) made using a reflector (or translucent body) of a specific shape (eg, inverted pyramid shape, cylindrical shape, etc.). It is possible to three-dimensionally output (display) an image (or a video) using an avatar (eg, a person, an animal, etc.) of a shape, and, depending on the type of the hologram device 100 , as shown in FIG. 6 , It is also possible to output a holographic image in the absence of a medium.

In addition, the terminal device 200 according to this embodiment physically and electrically (eg, wired, wireless, etc.) couple the hologram device 100 through the hologram device coupling unit 210, and a predetermined event situation may have occurred. When the hologram data corresponding to this event situation is transmitted to the hologram device 100, it is output.

In this case, the hologram device 100 is formed on one side (eg, left, center, right, etc.) of the terminal device operation unit (eg, a panel on which at least one of a button, a lever, and a jog dial is installed), so that the user can operate the terminal device. Make it easy to see while you're at it. In addition, as shown in FIG. 6 , the hologram apparatus 100 may be installed at a location capable of outputting a hologram image at the center of the screen or at the top of the screen according to the user's gaze (or face direction).

4 is a flowchart illustrating a method of controlling a terminal device for outputting a holographic image according to an embodiment of the present invention.

Referring to FIG. 4 , the controller 230 checks whether the hologram device 100 is coupled to the terminal device 200 ( S101 ).

When the hologram device 100 is coupled to the terminal device 200 according to the check result (Yes in S101), the control unit 230 detects the type information of the hologram device 100 through the hologram device type detection unit 220 . is detected (S102).

For example, as described above, the hologram device type detection unit 220 detects type information such as the model name, manufacturer, power supply standard, communication standard, resolution of an image (movie) to be reproduced, and audio standard of the hologram device 100 . do. The type information of the hologram device 100 may be directly provided by the hologram device 100 through a designated communication method, or may be input through a user with reference to published catalog information.

Thereafter, the control unit 230 checks whether a predetermined specific event for outputting hologram data occurs ( S103 ).

Accordingly, when a predetermined specific event (eg, program execution completion, information reception, information output, etc.) for outputting hologram data occurs (Yes in S103), the controller 230 controls the data to be output according to the event (hologram data). ) is converted to match the type of the hologram device 100 coupled to the terminal device 200 (S104).

In this case, the format conversion of the hologram data may be performed directly by the control unit 230 or may be performed through a separate hologram adapter unit 240 .

When the data format is changed (converted) as described above, the changed (converted) hologram data is transferred to the hologram device 100 connected (or coupled) to the hologram device coupling unit 210 (S105), and the The hologram device 100 outputs the hologram data to satisfy the user's curiosity and increase the fun of the terminal device user.

5 is an exemplary view showing a schematic configuration of a terminal device for outputting a holographic image according to a third embodiment of the present invention, and FIG. 6 is a view showing the holographic image output in various directions according to the user's gaze in FIG. 5 . It is an example diagram to explain how to do it.

Referring to FIG. 5 , the terminal device for outputting a holographic image according to the third embodiment of the present invention further includes a gaze tracking unit 250 for detecting and tracking a gaze (or face direction) of a user. At this time, in order to support the eye tracking unit 250 to detect and track the direction in which the user's eyes or face is facing, the terminal device includes a camera sensor (not shown) or an infrared sensor (not shown). can do.

At this time, although not shown in the drawings, the eye tracking unit 250 may be implemented by being included in the control unit 230 . That is, the control unit 230 including the eye tracking unit 250 controls the user's eye (ie, the line of sight) to be directed based on the information detected through the camera sensor (not shown) or the infrared sensor (not shown). It can detect and track the direction or the direction the face is facing.

As described above, when a designated event occurs in the terminal device 200 after detecting and tracking the direction in which the user's eyes (ie, gaze) or face is directed, the control unit 230 is shown in FIG. 6 . As described above, the hologram image is output (or expressed) through the hologram device 100 corresponding to the direction in which the user's gaze (or face direction) is directed.

Hereinafter, a method of outputting (or expressing) a hologram image in response to the user's gaze (or face direction) will be described with reference to FIGS. 6 and 7 .

7 is a flowchart illustrating a method of outputting a hologram image in various directions according to a user's gaze in FIG. 5 .

Referring to FIG. 7 , the control unit 230 detects and tracks the user's gaze (face direction) while manipulating the terminal device (S201).

Accordingly, when the user's gaze (face direction) is directed downward (ie, in the direction of the operation unit) of the preset first reference area (eg, a certain area at the center of the screen) (Yes in S202), the controller 230 controls the holographic image is output from the bottom (ie, the manipulation unit) to the top (ie, the direction corresponding to the user's gaze) (S203) (refer to FIG. 6(a) ).

Meanwhile, when the user's gaze (face direction) is directed toward a preset first reference area (eg, a certain area of the center of the screen) (Yes in S204), the control unit 230 displays the holographic image in the upper and lower portions of the first reference area. to the regions simultaneously (or sequentially and continuously) (S205) (refer to FIG. 6(c)).

In addition, when the user's gaze (face direction) is directed toward the top (ie, the top of the screen) of the preset first reference area (eg, a certain area of the center of the screen) (Yes in S206), the controller 230 displays the holographic image. Outputs from the top (ie, the manipulation unit) to the bottom (ie, the direction corresponding to the user's gaze) (S207) (refer to (b) of FIG. 6).

And although not shown in the drawing, when the user's gaze (face direction) is facing higher than the terminal device 200 (eg, the ceiling of the place where the terminal device is installed) (eg, looking at the ceiling or thinking ), a hologram image may be output in the air higher than the terminal device 200 as shown in FIG. 6( d ).

8 is an exemplary diagram showing a schematic configuration of a terminal device for outputting a holographic image according to a fourth embodiment of the present invention, and FIG. It is an example diagram for explaining a method of outputting.

Referring to FIG. 8 , the terminal device for outputting a holographic image according to the fourth embodiment of the present invention detects and tracks the movement of the user's hand (or arm) (ie, the direction and speed of the hand movement). It further includes a portion 260 . At this time, in order to support the hand tracking unit 260 to detect and track the movement of the user's hand (or arm) (that is, the direction and speed of the hand movement), the terminal device 200 includes a camera sensor (not shown). city) or an infrared sensor (not shown). Alternatively, an additional device that supports motion detection (eg, a device that transmits a specified specific signal such as RF, ultrasound, laser, or infrared) may be worn on the user's hand (or wrist).

At this time, although not shown in the drawings, the hand tracking unit 260 may be implemented by being included in the control unit 230 . That is, the control unit 230 including the hand tracking unit 260 detects the direction or speed of the user's hand movement based on information detected through the camera sensor (not shown) or an infrared sensor (not shown). and can be tracked.

Accordingly, when a designated event occurs in the terminal device 200, the controller 230 selects the hologram device 100 corresponding to the direction in which the user's gaze (or face direction) faces, as shown in FIG. 6 . Outputs (or displays) a holographic image through

And as described above, by detecting and tracking the movement of the user's hand, when the hand moves toward the displayed holographic image (eg, holding a virtual object expressed as a holographic image, shaking hands with a person expressed as a holographic image, or , when moving to select one of the virtual objects expressed as a holographic image), the control unit 230 controls an effect specified in response to the movement direction and speed of the hand in the holographic image that is already being output (eg, a hand-held effect). An effect of emphasizing a virtual object, an effect of waving a person's hand in a hologram, an effect of additionally displaying information related to a selected virtual object among hologram virtual objects, etc.) are applied and outputted (see FIG. 9 ).

Hereinafter, a method of applying and outputting (or expressing) a specified effect to an already output hologram image in response to the user's hand movement (the direction and speed of the hand movement) will be described with reference to FIGS. 9 and 10 .

FIG. 10 is a flowchart for explaining a method of changing and outputting hologram data according to the movement of a user's hand in FIG. 8 .

Referring to FIG. 10 , in a state in which an event has occurred in the terminal device 200 and a hologram image is already being output (or expressed) through the hologram device 100 ( S301 ), the control unit 230 controls the user's hand. The movement (ie, movement direction and speed) is detected and tracked (S302).

As a result of detecting and tracking the movement of the user's hand, when the hand moves toward the displayed holographic image (eg, an effect of highlighting and displaying a virtual object held in the hand, an effect of waving a hologram person's hand, a hologram virtual effect of additionally displaying information related to the selected virtual object among objects) (example of S303), the control unit 230 determines the effect specified in response to the direction and speed of the hand movement in the holographic image that is already being output (eg, : An effect of emphasizing and displaying a virtual object held in the hand, an effect of waving a hologram person's hand, an effect of additionally displaying information related to a selected virtual object among holographic virtual objects, etc.) is applied (S304).

At this time, the controller 230 outputs the hologram data to which the specified effect is applied in response to the user's gaze (or face direction) (S305) (refer to FIG. 9).

As described above, the present embodiment uses the hologram device to output a holographic image at a location corresponding to the user's gaze (or face direction) when a specified event occurs during manipulation of the terminal device, but the user's gaze (or face direction) ( It has the effect of improving the satisfaction of the user by outputting a more realistic holographic image by interacting with each other.

As described above, the present invention has been described with reference to the embodiment shown in the drawings, but this is merely exemplary, and various modifications and equivalent other embodiments are possible therefrom by those of ordinary skill in the art. will understand the point. Therefore, the technical protection scope of the present invention should be defined by the following claims. Implementations described herein may also be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementations of the discussed features may also be implemented in other forms (eg, as an apparatus or program). The apparatus may be implemented in suitable hardware, software and firmware, and the like. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a computer, a microprocessor, a processing device, including an integrated circuit or programmable logic device, or the like. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

100: hologram device 200: terminal device
210: hologram device coupling unit 220: hologram device type detection unit
230: control unit 240: hologram adapter unit
250: eye tracking unit 260: hand tracking unit

Claims (19)

a hologram device coupling unit formed in the terminal device to physically and electrically couple the plurality of hologram devices; and
While controlling the operation of the terminal device, when a predetermined specific event occurs in the terminal device in connection with the operation of the terminal device, hologram data to be output through the hologram device is generated or previously generated hologram data is called and a control unit for transmitting to the hologram device; including,
Further comprising; a hologram device type detection unit for detecting the type information of the hologram device coupled to the hologram device coupling unit;
The hologram device type detection unit,
Detects type information including all of the model name, manufacturer, power supply standard, communication standard, resolution of an image or video to be reproduced, and audio standard of the hologram device,
It is implemented so that the type information of the hologram device can be directly provided by the communication method specified by the hologram device, or the published catalog information can be inputted in a manual method,
The hologram device coupling unit,
While supplying power to the hologram device coupled to the terminal device, transmit data to be output to the hologram device coupled to the terminal device,
The control unit is
Detects and tracks the gaze of the user operating the terminal device,
When the user's eye tracking is directed in the direction of the manipulation unit below the first reference region corresponding to a predetermined region of the preset screen center of the terminal device, on one side of the manipulation unit formed under the first reference region In the hologram device coupled through the formed hologram device coupling unit, in response to the user's gaze, the hologram image is output from the lower portion where the operation unit is located to the upper direction in front of the user,
When the tracking user's gaze is directed in the direction of the preset first reference area of the terminal device, they are respectively coupled through a hologram device coupling unit formed on one side of the upper and lower operation units of the screen of the terminal device based on the first reference area. simultaneously outputting a holographic image in front of the user toward the first reference area in response to the user's gaze in the plurality of hologram devices,
When the eye of the tracking user is directed toward the upper side of the screen of the preset first reference area of the terminal device, the user in the hologram device coupled through the hologram device coupling unit formed on the upper portion of the screen with respect to the first reference area. In response to the gaze of the user, the hologram image is output in the downward direction in front of the user.
When the tracking user's gaze is directed toward the ceiling of the place where the terminal device is installed, which is higher than the screen of the terminal device, the user's gaze in the hologram device coupled through the hologram device coupling unit formed on one side of the upper screen of the terminal device A terminal device for outputting a holographic image corresponding to a movement event of a user's gaze and hand, characterized in that outputting a holographic image in an upward direction higher than that of the terminal device in front of the user in response to.
The method of claim 1,
Outputting a hologram image corresponding to a user's gaze and hand movement event, characterized in that it further comprises; a hologram adapter unit for converting the data format according to the type of the hologram device to transmit the hologram data to the hologram device terminal device.
According to claim 1, wherein the control unit,
Without separately forming a hologram adapter part,
A terminal device for outputting a holographic image corresponding to a movement event of a user's gaze and hand, characterized in that it is implemented so that a function of converting hologram data according to the type of the hologram device, which is a function of the hologram adapter unit, can be built-in.
According to claim 1, wherein the hologram device coupling portion,
As a position where a user can easily view an image or a moving image output from the hologram device even while operating a terminal device,
A holographic image corresponding to a user's gaze and hand movement event, characterized in that it is formed on at least one side of the left, right, and center of the operation panel of the terminal device in which at least one of a button, a lever, and a jog dial is installed. A terminal device that outputs .
delete delete The method of claim 1,
Further comprising; a gaze tracking unit for detecting and tracking the user's gaze or face direction;
The control unit is
When a specified event occurs on the terminal device,
A terminal for outputting a holographic image corresponding to a user's gaze and hand movement event, characterized in that the holographic image is output in a direction corresponding to the user's gaze through a hologram device positioned corresponding to the user's gaze or face direction Device.
delete delete The method of claim 1,
Includes; a hand tracking unit that detects and tracks the direction and speed of the user's hand movement;
The control unit is
It detects and tracks the direction and speed of the user's hand movement while the holographic image is already being output through the hologram device.
As a result of detecting and tracking the movement of the user's hand, when the hand moves toward the output holographic image, a predetermined effect is applied to the already output holographic image in response to the direction and speed of the hand movement,
A terminal device for outputting a holographic image corresponding to a user's gaze and hand movement event, characterized in that the hologram data to which the specified effect is applied is output in response to the user's gaze or face direction.
when a plurality of hologram devices are physically and electrically coupled to a terminal device for outputting a holographic image, the control unit checking whether a predetermined specific event for outputting hologram data occurs in the terminal device; and
When a predetermined specific event for outputting hologram data occurs, the controller transmits the specified hologram data to the plurality of hologram devices so that the hologram device outputs the hologram data;
When a plurality of hologram devices are coupled to the terminal device, detecting, by the controller, type information of the hologram device through a hologram device type detection unit;
The hologram device type detection unit,
Detects type information including all of the model name, manufacturer, power supply standard, communication standard, resolution of an image or video to be reproduced, and audio standard of the hologram device,
It is implemented so that the type information of the hologram device can be directly provided by the hologram device through a designated communication method, or open catalog information can be inputted in a manual manner,
A hologram device coupling unit for coupling the hologram device to the terminal device,
Transmitting data to be output to the hologram device coupled to the terminal device while supplying power to the coupled hologram device,
The control unit is
Detects and tracks the gaze of the user operating the terminal device,
When the tracking user's gaze is directed toward the manipulation unit below the first reference area corresponding to a predetermined area of the preset screen center of the terminal device, the user's gaze is placed on one side of the manipulator formed under the first reference area. In the hologram device coupled through the formed hologram device coupling unit, in response to the user's gaze, the hologram image is output from the lower portion where the operation unit is located to the upper direction in front of the user,
When the user's eye tracking is directed toward the preset first reference region of the terminal device, the first reference region is coupled to each other through hologram device coupling units formed on one side of the upper and lower operation units of the terminal device. simultaneously outputting a holographic image in front of the user toward the first reference area in response to the user's gaze in the plurality of hologram devices,
When the eye of the tracking user is directed toward the upper side of the screen of the preset first reference area of the terminal device, the user in the hologram device coupled through the hologram device coupling unit formed on the upper part of the screen with respect to the first reference area. In response to the gaze of the user, the hologram image is output in the downward direction in front of the user.
When the tracking user's gaze is directed toward the ceiling of the place where the terminal device is installed, which is higher than the screen of the terminal device, the user's gaze in the hologram device coupled through the hologram device coupling unit formed on one side of the upper screen of the terminal device A method of controlling a terminal device for outputting a holographic image corresponding to a movement event of a user's gaze and hand, characterized in that outputting a holographic image in an upward direction higher than that of the terminal device in front of the user in response to.
The method of claim 11, wherein when a plurality of hologram devices are coupled to the terminal device,
converting, by the controller, a format of the hologram data to be output according to the event to a type of the hologram device when a predetermined specific event for outputting the hologram data occurs; and
Transmitting the hologram data converted according to the type of the hologram device to the hologram device; Control method of a terminal device for outputting a holographic image corresponding to the user's gaze and hand movement event, characterized in that it further comprises.
13. The method of claim 12,
The conversion of the format of the hologram data according to the type of the hologram device is performed directly by the control unit or through a separate hologram adapter unit. A method of controlling a terminal device that outputs a holographic image.
delete delete 12. The method of claim 11,
detecting and tracking, by the control unit, the user's gaze or face direction through the gaze tracking unit; and
When a specified event occurs in the terminal device, outputting, by the control unit, a holographic image toward a direction corresponding to the user's gaze through the hologram device at a position corresponding to the user's gaze or face direction; A control method of a terminal device for outputting a holographic image corresponding to a user's gaze and hand movement event.
delete delete 12. The method of claim 11,
detecting, by the controller, the direction and speed of the movement of the user's hand through the hand tracking unit and tracking; and
As a result of detecting and tracking the movement of the user's hand, when the hand moves toward the output holographic image, the control unit performs a predetermined effect in response to the direction and speed of the hand movement in the already output holographic image. and outputting the hologram data to which the specified effect is applied in response to the user's gaze or face direction by applying How to control the device.

Images