KR101805065B1 - Apparatus and method for selectively controlling contents using eye tracking - Google Patents

Abstract

The apparatus includes a communication unit that receives a plurality of first images reproduced through one or more media devices from the one or more media devices, a gaze tracking unit that tracks a gaze of a user, An image acquiring unit for acquiring, as a second image, a first image at a point facing the gaze among the plurality of first images, and an image having the highest degree of similarity to the second image among the plurality of first images And a controller for selecting a content related to the selected first image among contents related to the plurality of first images.

Description

[0001] APPARATUS AND METHOD FOR SELECTIVELY CONTROLLING CONTENTS USING EYE TRACKING [0002]

The present invention relates to an apparatus and a method for selecting one of a plurality of images using a user's gaze tracking and performing control on a selected image and related contents.

With the development of information and communication technology, various media devices are flooded. Such a media device can provide various kinds of contents that the user can perceive through the five senses to the user, and such contents can represent a visual image that can be recognized visually, a sound that can be perceived by the auditory, and the like. In recent years, with the development of technology, it has been generalized that a single user has a plurality of media devices, so that it is possible to operate a plurality of media devices at the same time. It is also possible to simultaneously reproduce two or more images from one media device through functions such as screen division.

On the other hand, when a video is played back, generally different types of content, for example, sound is synchronized with the corresponding video and played back together. Accordingly, when a plurality of images are simultaneously reproduced, a plurality of sounds synchronized with each of the plurality of images are also reproduced together. At this time, the user generally tries to focus on only one image and the sound synchronized with the image because of the limitation of the recognition ability of the person. In the case of the image, the user can concentrate by fixing the gaze to a specific image, but in the case of the sound, it is difficult for the user to selectively concentrate on one sound. Therefore, it is necessary to stop the reproduction of sounds other than the sound synchronized with the image the user is concentrating on.

However, it is very troublesome to turn on / off the sound by the user's manipulation in a situation where the user changes the image to be focused at a short time interval. Furthermore, if the user wishes to control the image he or she views or the content related to the image, it is necessary to select the image that the user views, among the plurality of images, ≪ / RTI >

Korean Patent Publication No. 10-2008-0047909 (disclosed on May 30, 2008)

SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and system for finding out a video that is a focus of a user without a conscious act of the user when a plurality of videos are simultaneously played back, To be able to do so.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. will be.

The apparatus includes a communication unit that receives a plurality of first images reproduced through one or more media devices from the one or more media devices, a gaze tracking unit that tracks a gaze of a user, An image acquiring unit for acquiring, as a second image, a first image at a point facing the gaze among the plurality of first images, and an image having the highest degree of similarity to the second image among the plurality of first images And a controller for selecting a content related to the selected first image among contents related to the plurality of first images.

In addition, the communication unit may receive content related to the plurality of first images from the one or more media devices, and the content control device may further include an output unit for outputting content related to the selected first image among the received contents .

The control unit may control the one or more media devices through the communication unit to reproduce only the content related to the selected first image among the contents related to the plurality of first images.

The content control apparatus may further include an input unit for receiving an input from the user to perform predetermined operations on the first image or the content related to the first image, So that the operation can be performed only on the content related to the selected first image or the selected first image.

The image acquiring unit may acquire the second image by photographing an area within a predetermined range including a point toward which the line of sight is directed, among the plurality of first images.

In addition, the content control apparatus may further include a wearable portion that can be worn on the user's face portion, and the visual line tracing portion and the image obtaining portion may be formed integrally with the wearable portion.

A method of controlling content using eye tracking according to an embodiment of the present invention includes receiving a plurality of first images reproduced through one or more media devices from the one or more media devices, The method comprising the steps of: acquiring, as a second image, a first image at a point facing the line of sight among a plurality of first images; selecting an image having the highest degree of similarity to the second image among the plurality of first images; And selecting the content related to the selected first image among the contents related to the first image of the selected first image.

The content control method may further include receiving content related to the plurality of first images from the one or more media devices and outputting content related to the selected first image among the received contents have.

The content control method may further include the step of controlling the one or more media devices to reproduce only content related to the selected first image among contents related to the plurality of first images.

The content control method may further include the steps of: receiving, from the user, an input capable of performing predetermined operations on contents related to the first image or the first image; and controlling the one or more media devices, And performing the operation only on the selected first image or the content related to the selected first image.

The acquiring of the second image may include acquiring the second image by photographing an area within a predetermined range including a point toward which the sight line is directed from among the plurality of first images.

Also, the content control method may be performed such that the sequential execution of each step is repeatedly performed at a predetermined time interval.

According to an embodiment of the present invention, the user's gaze is tracked, the captured image is captured in the tracked direction, and the captured image is compared with a plurality of images being reproduced through the media device. And can control the corresponding image or the related contents. Accordingly, only the content related to the image to be watched by the user can be selectively reproduced or subjected to the control without requiring a separate selection action of the user, so that the user's troubles and mistakes associated with the image selection can be prevented.

1 is a conceptual view of a content reproduction system according to an embodiment of the present invention.
2 is a block diagram of a content control apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating a content control method according to an exemplary embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

1 is a conceptual view of a content reproduction system according to an embodiment of the present invention. However, since the content reproduction system 10 of FIG. 1 is only an embodiment of the present invention, the concept of the present invention is not limited to FIG.

The content reproduction system 10 is a system for providing a plurality of images and content (representative example, sound) related to each of the plurality of images to the user 20 of the content reproduction system 10, And one or more media devices (200).

The content control apparatus 100 may be connected to one or more media devices 200 in a wired or wireless manner. The content control apparatus 100 is not limited to the shape of the body part 101 but may be provided on the user 20 in addition to the body part 101 for convenience of the user 20 and effective implementation of a line- And a wearing part 102 to be worn.

The one or more media devices 200 may be a device for reproducing a plurality of images and contents related to each of the plurality of images, and the plurality of images are hereinafter referred to as a first image. Therefore, the media device 200 basically includes an image display unit 210 for displaying a first image and a device (e.g., a speaker) for playing back content related to each of the first images. When the first image is reproduced through the image display unit 210 of the media device 200, the content (typically sound) matched with each first image is also synchronized with the first image and reproduced through the media device 200 . Each of the media devices 200 may reproduce only one image of the first image and related contents, but may also reproduce two or more images and related contents at a time through a screen division function or the like. The media device 200 may further include a communication module (not shown) for exchanging data with the content control apparatus 100 in a wired or wireless manner. The specific types of such wired or wireless communication are obvious to those skilled in the art and will not be described in further detail. Such a media device 200 may be, for example, a television, a smart phone, or a computer system having a monitor and a speaker together, a content output device (e.g., a device such as a speaker for outputting sound) And the like.

2 is a block diagram of a content control apparatus according to an embodiment of the present invention. 2 may include a communication unit 110, a gaze tracking unit 120, an image acquisition unit 130, a control unit (not shown), and a control unit 140, an input unit 150, and an output unit 160.

The communication unit 110 may receive a plurality of first images reproduced from the media device 200 from one or more media devices 200 and may receive content associated with each of the plurality of first images. The communication unit 110 may also transmit a signal for controlling the media device 200 based on the control of the control unit 140. [ The communication unit 110 may be implemented through a hardware interface such as a wired or wireless communication module for transmitting data to the outside. The content control apparatus 100 may be connected to the media device 200 through wired or wireless communication in order to exchange data with the media device 200 via the communication unit 110. [ The detailed implementation of such a communication method is obvious to those skilled in the art and will not be described in further detail.

The gaze tracking unit 120 can track the direction in which the user 20 looks, i.e., the gaze of the user 20. [ As the line-of-sight tracking method for the function of the line-of-sight-tracking unit 120, a video analysis method, a contact lens method, a sensor attaching method, or the like may be used, but the present invention is not limited thereto. The specific implementation method of the eye tracking technology is also obvious to those skilled in the art, so that detailed description will be omitted.

The image acquiring unit 130 may acquire, as a second image, a first image at a point where a line of sight of the user 20 is directed among a plurality of first images. According to an embodiment, the image acquisition unit 130 may acquire a second image by photographing an area within a certain range including a point where the user 20's gaze is directed, out of a plurality of first images. The image acquiring unit 130 may be implemented as a general camera. The image acquiring unit 130 may include a component for receiving information on a user's eyesight from the eye-tracking unit 120 and setting a shooting direction in the eye direction .

The gaze tracking unit 120 and the image obtaining unit 130 may be integrated with other components of the content control apparatus 100. However, as shown in FIG. 1, for convenience of the user, May be integrated with the wearer 102, which is provided so as to be worn as general glasses, near the eyes, and may be physically separated from other components of the content control apparatus 100. [ In this case, the components other than the eye-tracking unit 120 and the image-acquisition unit 130 among the components of the content control apparatus 100 are implemented in the body 101 and physically separated from the wearer 102 The wearer 102 includes a separate communication module (not shown) so that the eye tracking unit 120 and the image acquisition unit 130 can exchange data with the components of the body 101 can do. At this time, the communication unit 110 can take charge of a communication function on the body 101 side.

The control unit 140 can calculate the degree of similarity between the first image received from the media device 200 and the second image acquired by the image acquisition unit 130. [ Such similarity calculation can be implemented by a generally known image processing algorithm, so that detailed description will be omitted here. Then, the control unit 140 can select an image having the highest degree of similarity to the second image among the plurality of first images. That is, the first image selected through the above-described series of processes can be judged to be an image that the user actually looks at. Accordingly, the control unit 140 can select the content related to the selected first image. Such a content can be reproduced through the media device 200 in synchronization with each first image, and can be typically a sound reproduced together with an image.

When the first image and the related contents are selected, the controller 140 controls the media device 200 through the communication unit 110 so that only the content related to the first image selected from among the contents related to the first image is reproduced. Can be controlled. Specifically, the control unit 140 may synthesize a signal for reproducing only the content related to the selected first image, and transmit the combined signal to the media device 200 through the communication unit 110. [ Then, the media device 200 can reproduce only the content related to the selected first image and not reproduce the content related to each of the remaining first images. If the content is a sound synchronized with each of the first images, the user can only hear the sound that is synchronized with the selected first image he / she is viewing, i.e., a sound desired by the user.

Alternatively, the content control apparatus 100 may directly reproduce the content related to the first image by itself in place of the media device 200. In this case, the media device 200 may be set in a state in which the reproduction function of the content related to the first image is inactivated, and may transmit the content related to each of the first images to the content control device 100. The content control apparatus 100 can receive the content through the communication unit 110. [ When the control unit 140 of the content control apparatus 100 selects the first image as described above, the control unit 140 determines that only the content related to the selected first image among the received contents is output to the output unit 160). ≪ / RTI >

In addition, according to an embodiment of the present invention, the content control apparatus 100 may perform control to allow the media device 200 to perform a specific operation. This will be described together with an input unit 150 to be described later. The controller 140 may be implemented by a computing device including a microprocessor.

The input unit 150 may receive, from the user 20, an input capable of performing a predetermined operation on the first image or the content related to the first image. For example, the user 20 may input a command for enlarging a screen of the first image or increasing a volume of the sound synchronized with the first image through the input unit 150. [ The control unit 140 controls the media device 200 through the communication unit 110 so that only the first video selected by the control unit 140 or only the content related to the selected first video Can be performed. That is, the control unit 140 controls the display unit 140 to display a screen on the first image that the user 20 is actually focused on, or to increase the volume of the sound synchronized with the first image, based on the input of the input unit 150 While it is possible to prevent the above-described operation from being performed on other first images and sounds synchronized therewith. The input unit 150 may include a keyboard, a mouse, a touch screen, a key pad, or a touch pad. The input unit 150 may be implemented on a separate device that can be carried by the user 20 independent of the body 101 of the content control apparatus 100. [ In this case, the input unit 150 may include a separate communication module for communicating with the components implemented in the body 101 through the communication unit 110.

The output unit 160 may output content related to the first image selected by the control unit 140 among the contents received through the communication unit 110 under the control of the control unit 140. [ With this operation of the output unit 160, it is possible to reproduce only contents related to the image in which the user 20 actually concentrates and gazes, such as a sound synchronized with the image. The output unit 160 may include a display device such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) or an audio output device such as a speaker.

3 is a flowchart illustrating a content control method according to an exemplary embodiment of the present invention. Each of the steps of the content control method performed through the content control apparatus 100 will be described with reference to FIG. 3 as follows. Descriptions overlapping with those of FIG. 1 and FIG. 2 may be omitted. Further, it goes without saying that the following steps are not necessarily performed in order, and that the order may be changed as necessary.

First, the communication unit 110 may receive a plurality of first images reproduced through the media device 200 from one or more media devices 200 (S110). When the gaze tracking unit 120 tracks the gaze of the user 20 of the content reproduction system 10 at step S120, the image obtaining unit 130 obtains the first image at the point of the user's gaze As a second image (S130). According to an embodiment, the image acquisition unit 130 may acquire a second image by photographing an area within a certain range including a point where the user 20's line of sight is directed, from among a plurality of first images.

Next, the control unit 140 can calculate the degree of similarity between the first image and the second image (S140). Thus, the control unit 140 calculates the degree of similarity between the first image and the second image, (S150). If an image having the highest similarity is selected from among the first images, the control unit 140 controls other components of the content control apparatus 100 so that only the content related to the selected image is reproduced (S160).

The steps of S160 may be performed in various manners. For example, the control unit 140 receives content related to each of the first images through the communication unit 110, and outputs the content related to the selected first image through the output unit 160 . On the other hand, the control unit 140 may control one or more media devices 200 so that only the content related to the first image selected from the content related to the first image is played through the media device 200 . That is, the subject of reproduction of the content related to the selected first image may be the content control apparatus 100, but may be one or more media devices 200, and other contents may not be reproduced at this time. As described above, in order for the step S160 to be performed in various manners, the controller 140 needs to select the content related to the first image selected through the step S150.

By performing each of the steps described above, it is possible to automatically find out an image taken by the user 20, and to selectively reproduce only the content related to the image. For example, when one or more media devices 200 reproduce a sound synchronized with each of a plurality of first images and a plurality of first images, the user 20 may select a sound other than a sound synchronized with the first image You may want to turn off the sound. In this case, it is very troublesome and a mistake may arise if manually turning the sound off and on manually. However, according to the content control method according to the embodiment of the present invention, the content control apparatus 100 Thus, the convenience of the user 20 can be improved, and a mistake of the user 20, such as in a manual operation, can be prevented.

If the input unit 150 receives an input from the user 20 that allows the user to perform predetermined operations on the first image or the content related to the first image during the execution of the steps described above (S170) The control unit 140 may control one or more media devices 200 to perform the predetermined operation only on the selected first image or the content related to the selected first image in operation S180.

By performing the steps up to S170 and S180, it is possible to automatically obtain an image to be viewed by the user 20 and to control the image or related contents. For example, the user 20 may desire to enlarge only the image of the first image reproduced by the media device 200, which he / she is concentrating on, or may want to increase the volume of the sound synchronized with the image. In this case, according to the content control method of the present invention, since the user 20 can perform the screen enlargement or volume increase only by omitting the process of selecting the image he / she is viewing, And it is also possible to prevent a mistake of the user 20 such as in manual selection.

Meanwhile, the process of sequentially performing the steps S110 to S180 may be repeatedly performed at predetermined time intervals. Accordingly, the user 20 can fully enjoy the advantages of the content control method according to the embodiment of the present invention during the repetitive execution.

Combinations of each step of the flowchart and each block of the block diagrams appended to the present invention may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and changes may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as falling within the scope of the present invention.

10: Content playback system
20: User
100: Content control device
101:
102:
110:
120: eye tracking unit
130:
140:
150:
160: Output section
200: Media device
210:

Claims (14)

A communication unit for receiving a plurality of first images reproduced through one or more media devices from the one or more media devices;
A gaze tracking unit for tracking a user's gaze;
An image acquiring unit that acquires, as a second image, a first image at a point facing the gaze, out of the plurality of first images; And
And a controller for selecting an image having the highest degree of similarity to the second image among the plurality of first images and selecting a content related to the selected first image among contents related to the plurality of first images
Content control device using eye tracking. The method according to claim 1,
Wherein the communication unit receives content related to the plurality of first images from the one or more media devices,
The content control apparatus may further include an output unit for outputting content related to the selected first image among the received contents
Content control device using eye tracking. The method according to claim 1,
Wherein the control unit controls the one or more media devices through the communication unit to reproduce only the content related to the selected first image among the contents related to the plurality of first images
Content control device using eye tracking. The method according to claim 1,
Further comprising an input unit for receiving an input from the user that enables predetermined operations to be performed on contents related to the first image or the first image,
The control unit synthesizes a signal for performing the operation only on the content related to the selected first video or the selected first video based on the input and transmits the synthesized signal to the media device through the communication unit
Content control device using eye tracking. The method according to claim 1,
Wherein the image acquiring unit acquires the second image by photographing an area within a predetermined range including a point toward which the line of sight is directed from among the plurality of first images
Content control device using eye tracking. The method according to claim 1,
Further comprising a wearable portion that can be worn on the face of the user,
Wherein the visual line tracing unit and the image obtaining unit are formed integrally with the wearer
Content control device using eye tracking. Receiving from the one or more media devices a plurality of first images to be played through one or more media devices;
Tracking a user's gaze;
Acquiring, as a second image, a first image at a point facing the line of sight among the plurality of first images;
Selecting an image having the highest degree of similarity with the second image among the plurality of first images; And
Selecting a content associated with the selected first image from among content associated with the plurality of first images
Content Control Method Using Eye Tracking. 8. The method of claim 7,
Receiving content associated with the plurality of first images from the one or more media devices; And
And outputting the content related to the selected first image among the received contents
Content Control Method Using Eye Tracking. 8. The method of claim 7,
Controlling the one or more media devices to reproduce only the content related to the selected first image among the contents related to the plurality of first images
Content Control Method Using Eye Tracking. 8. The method of claim 7,
Receiving, from the user, an input capable of causing a predetermined operation to be performed on the first image or the content related to the first image; And
And synthesizing a signal for performing the operation only on the content related to the selected first image or the selected first image based on the input, and transmitting the synthesized signal to the media device
Content Control Method Using Eye Tracking. 8. The method of claim 7,
Wherein the acquiring of the second image includes capturing an area within a predetermined range including a point toward which the visual line is directed from among the plurality of first images to acquire the second image
Content Control Method Using Eye Tracking. 8. The method of claim 7,
The sequential execution of each of the steps is repeatedly performed at predetermined time intervals
Content Control Method Using Eye Tracking. Receiving from the one or more media devices a plurality of first images to be played through one or more media devices;
Tracking a user's gaze;
Acquiring, as a second image, a first image at a point facing the line of sight among the plurality of first images;
Selecting an image having the highest degree of similarity with the second image among the plurality of first images; And
Selecting a content related to the selected first image among the contents related to the plurality of first images
A program stored on a computer readable recording medium. Receiving from the one or more media devices a plurality of first images to be played through one or more media devices;
Tracking a user's gaze;
Acquiring, as a second image, a first image at a point facing the line of sight among the plurality of first images;
Selecting an image having the highest degree of similarity with the second image among the plurality of first images; And
Selecting a content related to the selected first image among contents related to the plurality of first images;
Computer readable recording medium.

Images