KR101585466B1 - Method for Controlling Operation of Electronic Appliance Using Motion Detection and Electronic Appliance Employing the Same - Google Patents

Abstract

An electronic device operation control method that enables a user to estimate a point on a screen intended to be pointed by a user quickly, accurately, and stably while tracing a user's operation, thereby enabling a user to effectively and conveniently adjust functions of the electronic device; , Which employs it to enable a user to adjust functions effectively and conveniently. The operation control method of the present invention is implemented in an electronic device having a display screen and an image sensor, and allows a user to operate the electronic device without touching the electronic device. According to a preferred embodiment, first, an image is captured in front of the display screen using an image sensor. And determines the position of the user's forefinger end point and the near center point in the captured image. Then, a point at which a virtual straight line connecting the center point of the forehead and the forefinger end meets the display surface is estimated as a point at which the user wants to point, and a pointer is displayed at the estimated pointing point. The user of the electronic device can accurately and stably point the menu on the screen just like using the touch panel without touching the electronic device directly or using the remote controller, and the convenience of the electronic device is greatly increased.

Television, user interface, UI, touch screen, pointing, camera, image sensor

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method for controlling an operation of an electronic device by motion detection,

More particularly, the present invention relates to an electronic device that detects a shape or motion of a user and estimates an operation command intended by a user from the detected electronic device, and a method of controlling an operation of such an electronic device .

Thanks to the development of communication technology and component technology, television has made great progress in various aspects such as full color screen, adoption of digital transmission method, diversification of transmission channel, and enlargement of screen. However, in spite of the breakthrough in essential functions such as broadcast signal transmission and image reproduction, it is a reality that the operation method of the user interface is very slow to develop. That is, a television user can adjust the television function only by pressing a button provided on the outside of the receiver or by operating the remote control, as in the case of 20 to 30 years ago. Accordingly, when the remote controller can not be found or the battery of the remote controller is discharged, the user must move to the television main body and perform button operation. However, if the television installation place is wide, this operation is not troublesome.

In order to solve such a problem, Japanese Laid-Open Patent Publication No. 1997-57020 (name of the invention: television control apparatus based on speech recognition) recognizes the user's voice and displays the OSD information for the selection menu on the screen, So that the convenience of use can be achieved even when the remote controller can not be easily found. However, the television operating on the basis of speech recognition as described above can not only be awkward in instruction commands by voice, but also can cause the user to be careful because the normal conversation of the user can be confused with the television operation control command. .

Various attempts have been made to detect the shape or motion of the user instead of the voice, and to recognize and operate the operation command therefrom.

For example, in Japanese Laid-Open Patent Publication No. 2003-21988 (titled remote finger manipulator using image processing), a user's hand is photographed with a camera attached to a television receiver to recognize a pattern of a user's finger, And a method for performing the corresponding function when the stored pattern matches the pattern for performing the specific function. In addition, Japanese Laid-Open Patent Application No. 2007-103895 (entitled " hand gesture recognition system and method ") discloses a hand gesture recognition system and method that can recognize the hand gesture of a user in an image captured by a camera, System is described. However, since the television according to the above-described method can recognize only some gestures defined in advance, in order to perform a specific function desired by the user, it is troublesome to learn and learn the predetermined gesture pattern in advance in order to perform the function And it is impossible to point to a specific point on the screen.

Open No. 2003-37692 (a remote control system using hand direction recognition), a person's shape is acquired as image data by a plurality of cameras, and the shape of the hand from the shape of the person obtained and the position of the hand And a direction in which the hand points are indicated, thereby controlling the electronic apparatus. However, in order to implement this system, not only a plurality of cameras need to be installed indoors, but there is still a problem that pointing by hands is substantially difficult when the direction pointed by the user's hand is unstable.

On the other hand, ideas for operating the device by detecting the position and the eyes of the eyes are also presented.

For example, Japanese Unexamined Patent Publication (Kokai) No. 2000-56563 (entitled position registration system) acquires a face image of a user through a camera and calculates a user's gazing position from the obtained digital video signal, And a control unit for controlling the control unit. However, this system has a problem that the computational power burden is great in obtaining the face shape and calculating the gazing position, and it is difficult to precisely point the gaze when the gaze position is incorrect.

Open Publication No. 2005-120132 (titled "remote control television by user motion using image processing and control method thereof") detects a user motion by a camera, and when a specific part of the user's body is within a predetermined time range The virtual menu is activated when a predetermined number of times exist at the position of the user's eyes, the virtual menu is displayed at a position facing the user's eyes by recognizing the position of the user's eyes, A system for executing functions is described. However, in this system, since the user recognizes the menu to be selected by the position of the user's hand in the captured image, when the screen of the television is large and the distance between the screen and the user is close to each other, Furthermore, since the user must keep his / her hands fixed for a certain period of time in order to select a menu, his / her arm is sick and menu selection and function execution may take a long time.

As described above, various methods for detecting the shape, movement, gaze, etc. of the user so as to solve the inconvenience of remote control operation have been proposed in various ways. However, the conventional methods are not suitable for pointing inaccurate or unstable And it takes a long time to select a menu, which makes it difficult to perform real-time processing. Due to such a problem, there is not yet an example in which a television adopting methods of detecting the shape, movement, sight line, etc. of the user and controlling the television operation is commercially performed at the time of the present application.

Pointing on television is a serious problem especially on Internet TV. The Internet TV is a TV having a function of receiving a television broadcasting signal and having an internet connection function. In such an Internet TV, it is very difficult to select one of a number of hypertext and menu items on a web document displayed on the screen. Although various types of remote controllers have been proposed for the purpose of facilitating web surfing, most of the remote controllers are only inconvenienced when viewing television broadcasts, while the inconvenience of web surfing is not improved greatly. That is, the keys mainly used in the outer surface of the main body or on the keypad of the remote controller for the user to operate the television at the time of watching television broadcast are 'channel up (+)', 'channel down (-)' , 'Volume down (-)', and 'power on / off'. However, the remote controllers for web surfing, which have increased the number of operation keys, do not increase the product cost, Weight.

Accordingly, there is a desperate need for a television user to be able to quickly, accurately, and stably point to a specific point on the screen while not directly touching the television while watching TV or surfing the Internet.

[Patent Document 1] Published Japanese Patent Application No. 2003-21988 2003. 3. 15.

[Patent Document 2] Published Japanese Patent Application No. 2007-103895 Published Oct. 25, 2007

[Patent Document 3] Published Unexamined Patent Application No. 2003-37692 Published on May 16, 2003

[Patent Document 4] Published Unexamined Patent Application No. 2000-56563 September 15, 2000 [

[Patent Document 5] Published Japanese Patent Application No. 2005-120132 2005. 12. 22.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to provide a method and apparatus for tracking an operation of a user, accurately and stably estimating a point on a screen on which a user intends to point, The present invention also provides an image display device that enables a user to effectively and conveniently adjust a function.

In addition, the present invention can estimate a point on the screen that the user intends to point to quickly, accurately, and stably while tracing the user's operation, thereby allowing the user to adjust the functions of the electronic device effectively and conveniently And to provide a method of controlling a device operation.

According to an aspect of the present invention, there is provided an image display apparatus including a display unit having a display surface, a video signal processing unit, an image capture unit, and a control unit. The video signal processing unit provides a video signal of a predetermined format to the display unit so that an image corresponding to the video signal is displayed on the display surface. The image capturing unit captures an image in front of the display surface. The control unit determines the position of the first reference point in the user's hand and the second reference point in the user's body in front of the display surface using the image captured by the image capture unit, To estimate a point the user wants to point to on the display surface. The control unit controls the video signal processor to display a predetermined pointer on the estimated pointing point on the display surface of the display unit.

In the present specification including the claims, the term "body" is used to refer to a portion of the user's body excluding an upper and lower hands under the elbow and a hand.

In a preferred embodiment, the image display device is implemented in the form of a digital television. In this embodiment, the image display apparatus further includes a broadcast signal receiving unit for receiving a broadcast signal, demodulating and decoding the received broadcast signal, and outputting a video bit stream. At this time, the video signal processor converts the video bitstream into the format, and outputs the format-converted video signal to the display unit so that an image corresponding to the format-converted video signal is displayed on the display screen.

Preferably, the image capture unit has two or more image sensors disposed at different positions from each other in front of the image display device. Particularly preferably, the image capture unit is composed of two image sensors arranged such that the horizontal position and the vertical position are different from each other.

In a preferred embodiment, the video signal processor is configured to combine the format-converted video signal with predetermined on-screen display (OSD) adjustment menu data to provide a predetermined on-screen display adjustment to an image corresponding to the format- Menus are merged and displayed on the display together with the pointer.

In a preferred embodiment, the OSD adjustment menu data is provided to the video signal processor by the controller.

The first reference point may be a finger end point. As the second reference point, an eyeball center point of the one eye (eye), a midpoint between the eyeball centers, that is, one point on the shoulder contour line, the upper end point of the head, and the elbow point .

In a preferred embodiment, the image display apparatus further comprises selection event detection means for detecting a predetermined selection event from the image captured by the image capture unit. The control unit may control the image display device to perform an operation corresponding to the pointing point in response to the selection event.

In order to solve the above technical problems in a more general aspect, the present invention provides an electronic apparatus having a display unit having a display surface, an image capture unit, and a control unit. The image capture unit is installed on the front surface of the electronic device, and captures an image on the front of the display surface. The control unit determines the position of the first reference point in the user's hand and the second reference point in the body using the image captured by the image capture unit and uses the position information of the first and second reference points, Estimates a point to be pointed on the display surface, and displays a predetermined pointer on the estimated pointing point on the display surface of the display unit.

Preferably, the image capture unit comprises two or more image sensors disposed at different positions on the front surface of the electronic device. More preferably, the image capture unit is composed of two image sensors arranged such that the horizontal position and the vertical position are different from each other.

According to another aspect of the present invention, there is provided an electronic device operation control method of an electronic device including a display screen and an image sensor, wherein the electronic device is operated in a state in which the user does not touch the electronic device I can do it. First, the front of the display surface is imaged using the image sensor. Determines the position of the first reference point in the user's hand in the captured image, and determines the position of the second reference point in the user's body. Then, using the position information of the first and second reference points, the user estimates a point to be pointed on the display surface. Finally, a pointer is displayed at the estimated pointing point.

The first reference point may be a finger end point. As the second reference point, an eyeball center point of the one eye (eye), a midpoint between the eyeball centers, that is, one point on the shoulder contour line, the upper end point of the head, and the elbow point .

In estimating the pointing point, a point at which a virtual straight line connecting the first reference point and the second reference point meet the display surface can be estimated as the pointing point. However, the phenomenon that the user's hand hides part of the display surface is reduced, and the estimation method may be modified such that the user does not need to raise the arm. As an example, it is possible to determine an auxiliary reference point that is a predetermined distance from the first reference point, and to estimate a point where a virtual straight line connecting the auxiliary reference point and the second reference point meets the display surface. As another example, it is possible to determine an auxiliary reference point that is a predetermined distance below the second reference point, and to estimate a point where a virtual straight line connecting the first reference point and the auxiliary reference point meets the display surface as a pointing point . As another example, it is possible to determine a point at which a virtual straight line connecting the first reference point and the second reference point meet with the display surface, and to estimate a point that is a predetermined distance from the determined point as a pointing point.

It is preferable to monitor whether a predetermined selection event is generated by the image sensor in a state where a pointer is displayed on a display display surface and to perform an operation corresponding to a pointing point in response to the selection event.

In addition, it is preferable to monitor whether a predetermined function inactivation event occurs, and to erase the pointer on the display surface when the function inactivation event occurs, so that the pointer is selectively displayed only when necessary. The function deactivation event may be, for example, that the time when the pointing point is outside the display surface continues for a predetermined time or that the pointer or the first reference point is stopped for a predetermined time or more.

According to the present invention, an electronic device user can precisely and stably point the on-screen menu, such as using a touch panel without touching the electronic device directly or using a remote control, and can perform pointing, menu / It is possible to process the function of user's intention in real time such as function adjustment. Accordingly, an intuitive user interface for an electronic device becomes possible, and in particular, the user's prior learning is hardly required in realizing such a user interface. Therefore, there is an effect that the convenience of the electronic device is greatly increased.

FIG. 1 conceptually shows an example in which a user points at a point on a screen in a digital television receiver according to the present invention.

A liquid crystal panel display 50 is provided on the front surface of the digital television receiver and the display surface 52 of the display 50 is arranged in a vertical direction so as to face forward. On the other hand, image sensors 54a and 54b are provided in the vicinity of two corner points located diagonally from the front face of the housing for housing and protecting the four corners of the liquid crystal panel display 50.

When a user wishes to adjust a volume, channel, or other function while viewing a broadcast through such a digital television receiver, the user may select a specific gesture predetermined, and thus, a halftone or full tone OSD (On-screen Display) adjustment menu to be superimposed on the broadcast image. In one embodiment, the OSD adjustment menu includes a 'channel up (+)' icon 56a, a 'channel down (-)' icon 56b, an 'up' icon 56c, - 'icon 56d, a volume level indicator 56e, and a' power on / off 'icon 56f.

With the OSD control menu displayed, the user can point to this position by pointing his or her hand at the position he or she wishes to point, without using a separate remote control or pointing device. In one embodiment, the television receiver captures the frontward center point Peye (Xeye, Yeye, Zeye) of the user and the finger end point Pf (Xf, Yf, Zf). Here, the finger may be a finger or a forefinger extending from the user. The receiver then determines a point Ppoint (Xpoint, Ypoint, Zpoint) at which a virtual straight line connecting the two reference points (Xeye, Yeye, Zeye), (Xf, Yf, Zf) meets the display screen 52, It is assumed that the point is intended to be pointed by the user. Then, the digital television receiver displays the pointer at the estimated pointing point and emphasizes and displays the icon positioned at the pointer point.

The pointing point can be determined quickly because the coordinates of the two reference points Peye (Xeye, Yeye, Zeye) and Pf (Xf, Yf, Zf) can be easily determined even if there is accuracy variation according to the user's attitude or the like . Also, since the two points are spaced by 20 to 30 centimeters (cm), the stability of the pointing is very high. When the user moves the hand, the pointer on the display surface quickly follows the movement of the hand. This pointing, which is intuitive and sensible, is not only easy, but also simple because it does not require any device manipulation. Thus, by pointing the finger in the direction of the receiver, the user can easily and easily point a point on the display surface 50 of the display 50 rather than using the touch panel.

On the other hand, in a state in which the pointer is displayed at a point where the pointing was intended, the user directs the predetermined selection event, for example, by bending the forefinger forward in the same manner as pressing the forward virtual button, Selected, and executed. For example, if a user brings a selection event while holding a pointer on a 'channel up (+)' icon, the receiver increases the broadcast signal receiving channel number. When the selection event occurs continuously, the channel number can be continuously increased correspondingly. The volume can also be changed in the same way. On the other hand, instead of sequentially selecting the 'volume up (+)' or 'volume down (-)' icon, the volume level can be changed in several steps at a time by selecting a specific segment on the level indicator 56e.

Figures 2a and 2b show a coordinate system used to determine the coordinates of two reference points Peye (Xeye, Yeye, Zeye), Pf (Xf, Yf, Zf) in one embodiment. In the present embodiment, the center point of the display display surface 52 is used as the origin for determining the position of each point Po (Xo, Yo, Zo) in the front space of the receiver. Then, coordinate axes of the orthogonal coordinate system are appropriately arranged on the basis of this origin.

3 shows an embodiment of a digital television receiver according to the present invention. The digital television receiver includes a broadcast signal receiving unit 10, a video signal processing unit 40, a display 50, and a system control unit 60.

In the broadcast signal receiving unit 10, the tuner 12 selects a broadcast signal of a channel selected by a user from among a plurality of broadcast signals inputted through an antenna or cable, and outputs the selected broadcast signal. The demodulator 14 demodulates the broadcast signal from the tuner 12 and outputs the demodulated transport stream by a demodulation technique corresponding to the modulation scheme adopted in the receiver, for example, the 8-VSB modulation scheme. The demultiplexer 16 demultiplexes the demodulated transport stream to separate the encoded video bit stream and the encoded audio bit stream. The video decoding unit 18 decodes the encoded video bit stream and outputs a video bit stream.

The video signal processing unit 40 converts the video bit stream from the video decoding unit 18 into a video output format that can be displayed and the display 50 displays an image corresponding to the format converted video signal on the display surface 52, . In addition to the broadcast image, the OSD adjustment menu for function adjustment may be displayed as a half tone or a full tone on the display screen 52 of the display 50. On the other hand, in addition to the broadcast video and OSD adjustment menu, the Internet content may be displayed on the display 50 (52). To this end, the video signal processing unit 40 receives the selection control signal, the OSD display data or the Internet content data from the system control unit 60, and outputs the OSD adjustment menu to the format- And provides the video or Internet content video signal to the display 50. The display 50 displays an image corresponding to the video signal from the video signal processing unit 40. [

The speech decoding unit 20 decodes the encoded audio bit stream and outputs an audio bit stream. A digital-to-analog converter (DAC, not shown) converts the audio bitstream to an analog audio signal, and an amplifier (not shown) amplifies the analog audio signal and outputs it through the speaker 30.

The system control unit 60 controls each part of the receiver as a whole and adjusts the function of each part of the receiver in response to a user's operation command. The system control unit 60 is implemented by a microprocessor or a microcontroller, and in particular, may be realized by two or more microprocessors or microcontrollers whose roles are shared. The system control unit 60 can receive a user's operation command from buttons (not shown) or a remote controller provided on the outer surface of the receiver main body.

Further, according to the present invention, the system control unit 60 estimates a point on the display surface on which the user intends to point, from the user image captured by the image sensors 54a and 54b, have. When the pointing point is estimated, the system control unit 60 provides the video signal processing unit 40 with the video signal related to the pointer so that the pointer is displayed at the pointing point of the display surface of the display 50. In this state, when the user selects a specific function or menu, the system control unit 60 controls the corresponding part of the receiver so that the corresponding function adjustment is made in response to the selection.

In the memory 70, the ROM 72 stores an operating system or firmware for operation of the system control unit 60, and an application program necessary for implementing the present invention. In particular, the ROM 72 preferably stores user gesture pattern data necessary for judging activation of the control function according to the present invention and contour pattern data required for finger end point extraction. The RAM 74 stores temporary data generated during the system operation. All or a part of the memory 70 may be replaced by a built-in memory of the microcontroller constituting the system control unit 60. [

The network interface 80 allows the system control unit 60 to access the Internet. The network interface 80 may be, for example, a cable modem, an xDSL modem, an optical network unit (ONU), or a fiber-to-the-home (FTTH) modem. Meanwhile, although the network interface 80 may be embedded in the digital television receiver as described above, in a modified embodiment, a modem of a type separately made from the digital television receiver may be externally connected to the digital television receiver. The system control unit 60 accesses an external web server, a mail server, or an FTP server via the Internet in accordance with a user control command to request contents and transmits the received contents to the display 50 through the video signal processing unit 40 Can be displayed. Here, the Internet-related control commands such as the Internet function selection, the Internet access request, the HTTP request, the E-mail reception / FTP request, etc. are also displayed on the display 50, It is needless to say that it is possible to estimate the pointing point intended by the user in the captured image and to recognize and accept the selected event.

FIG. 4 shows a configuration of a program module executed by the system control unit 60. FIG. In addition to the main control unit 100 for controlling the operation of each program module, the function activation event detection unit 102, the reference point pixel detection unit 104, the pointer position determination unit 106, the pointer display control unit 108, The operation control unit 112, and the function inactivation event detection unit 114 are executed in the system control unit 60. [

4 does not constitute a part of the system control unit 60, but the first to fourth frame memories 122 to 128 are shown together for convenience of explanation. Each of the first to fourth frame memories 122 to 128 stores image data captured by the image sensor 54a or 54b in units of one frame. The first frame memory 122 stores the image data of the current frame captured by the first image sensor 54a. The second frame memory 124 stores the image data of the previous frame captured by the first image sensor 54a. Here, instead of using all of the image frames sensed by the image sensors 54a or 54b, it is possible to skip a part of the image frames and to select only one frame per 0.1 second or 0.5 second in order to reduce the computation burden. In this case, 'previous frame' refers to a frame 0.1 seconds or 0.5 seconds before. The 3-frame memory 126 stores image data of the current frame captured by the second image sensor 54a. The fourth frame memory 128 stores image data of a previous frame captured by the second image sensor 54a. In one embodiment, the first to fourth frame memories 122 to 128 may be implemented by the RAM 74 of the memory 70.

In the system control unit 60, the main control unit 100 causes the function activation event detection unit 102 to monitor whether an event for activating the control function according to the present invention occurs. When an event occurrence report or a control function activation request is received from the function activation event detection unit 102, the main control unit 100 displays the OSD control menu on the display 50 as shown in FIG. 1, And controls the video signal processing unit 40 through the display control unit 108. In response to the control function activation request, the main control unit 100 activates the reference point pixel detection unit 104, the pointer position determination unit 106, the pointer display control unit 108, and the selection event detection unit 110.

The main control unit 100 causes the reference point pixel detection unit 104 to detect the pixel position of the reference points in the image acquired by the image sensors 54a and 54b and causes the pointer position detection unit 106 to detect the pixel position Based on the positions, the user determines the pointer position by estimating the position of the point to be pointed. Upon receiving the pointer position, the main control unit 100 causes the pointer display control unit 108 to readjust the pointer position and highlight the icon around the pointer. Upon reception of the selection event detection signal from the selection event detection unit 110, the main control unit 100 controls the tuner 12, the audio amplifier (not shown) through the operation control unit 112 to perform the function adjustment corresponding to the icon around the pointer, And the like). The main control unit 100 controls the pointer display control unit 108 to erase the OSD adjustment menu on the display 50. The control unit 100 controls the reference point pixel detection unit 104, The pointer position determination unit 106, the pointer display control unit 108, and the selection event detection unit 110 to the standby mode.

The function activation event detection unit 102 monitors whether a predetermined function activation event is generated as a control function activation condition according to the present invention. When the function activation event occurs, the main control unit 102 performs a process for activating the control function And execute it. In a preferred embodiment, the function activation event is determined by the user taking a gesture of a predetermined pattern for a period of time, e.g., 3 seconds or more. Examples of such gestures include winking with one eye, holding a palm or fist at a specific area of the face, or pointing a finger or palm toward the monitor.

The function activation event detection unit 102 reads out the image data stored in the first frame memory 122 every predetermined period (for example, one second) and compares the image data with the gesture pattern data stored in the ROM 72, And judges whether the gesture is taken. If it is determined that the user has taken the gesture and remains for a predetermined time or more, the function activation event detection unit 102 determines that the user requests the function activation and requests the main control unit 100 to activate the control function.

The reference point pixel detection unit 104 reads the image data of the current frame captured by the first image sensor 54a from the first frame memory 122 and outputs the outline of the user, which is an object in this image (154a in Fig. 5) And the position of the pixel (155a in Fig. 5) corresponding to the first reference point in the image is determined by comparing this contour with the pattern stored in the memory 70. Fig. Here, the first reference point refers to a finger end point of a hand, particularly a stretched finger or an index finger of a hand, near the screen of both hands of the user's body. Various methods for contour detection and pattern comparison have been proposed, and a detailed description thereof will be omitted because those skilled in the art can easily implement the present invention.

Similarly, the reference point pixel detection section 104 reads the image data of the current frame captured by the second image sensor 54b from the third frame memory 126, The position of the pixel (155b in Fig. 5) corresponding to one reference point is determined. Further, the reference point pixel detection unit 104 determines the positions of pixels corresponding to the left and right eyeball centers of the user in the images (154a and 154b in Fig. 5) respectively captured by the first and second image sensors 54a and 54b do. The positions of the pixels corresponding to the centers of the left and right eyeballs in each image are averaged to determine the positions of the pixels corresponding to the second reference point, that is, the center of gravity of the left and right eyeball centers.

The pointer positioning unit 106 determines the positions of the pixels corresponding to the first and second reference points in the image (154a, 154b in Fig. 5) respectively picked up by the first and second image sensors 54a, 54b 155a and 55b from the reference point pixel detection unit 104 and calculates the spatial coordinates of the first reference point Pf (Xf, Yf, Zf) and the second reference point Peye (Xeye, Yeye, Zeye) do. The pointer position determination unit 106 determines the pointer position on the display screen 52 using the spatial coordinates of the first reference point Pf (Xf, Yf, Zf) and the second reference point Peye (Xeye, Yeye, Zeye) Ppoint (Xpoint, Ypoint, 0) is determined.

For this purpose, the pointer positioning unit 106 firstly calculates the position of the pixel 155a determined to correspond to the first reference point, that is, the finger end point, with respect to the image 154a picked up by the first image sensor 54a An equation of a hypothetical straight line 156a describing the positions of the points at which images can be formed in the pixel is calculated. The pointer positioning unit 106 also determines the position of the pixel 155b determined to correspond to the first reference point in the image with respect to the image 154b picked up by the second image sensor 54b, An equation of a hypothetical straight line 156b describing the positions of points at which an image can be formed is calculated. Then, the pointer positioning unit 106 determines the intersection of two straight lines. The intersection determination may be performed by solving the equations of the two straight lines 156a and 156b in an algebraic manner and solving them. However, taking the calculation error into consideration, it is necessary to numerically analyze, for example, iteration to make the distance from the straight line It is desirable to determine nearby points and calculate their center point. This intersection becomes the first reference point Pf (Xf, Yf, Zf). Also, the second reference point Peye (Xeye, Yeye, Zeye), that is, the center-of-gravity point of the user, can be determined in a similar manner.

1, a virtual straight line connecting the two reference points Pf (Xf, Yf, Zf), Peye (Xeye, Yeye, Zeye) is displayed on the display display surface 52 The coordinates (Xpoint, Ypoint, 0) of the meeting point are determined, and this point is assumed to be a point that the user intends to point to.

The pointer display control unit 108 outputs the selection control signal to the video signal processing unit 40 in response to the menu activation instruction signal of the main control unit 100 and displays the OSD adjustment menu on the display 50 And provides the OSD adjustment menu data to the video signal processing unit 40 so as to be superimposed and displayed. As shown in FIG. 1, the OSD adjustment menu includes a 'channel up (+)' icon 56a, a 'channel down (-)' icon 56b, a ' Icon 56d, a volume level indicator 56e and a power on / off icon 56f, and further includes a receiver function adjustment icon or an 'OSD adjustment menu end' icon . When the pointer is displayed, the icon located at the point of the point is highlighted in color, brightness, or saturation. When the pointer coordinate data is directly received through the main control unit 100 or from the pointer positioning unit 106, the OSD adjustment menu data is updated in accordance with the new pointer coordinates and provided to the video signal processing unit 40. [

The selection event detection unit 110 monitors whether a selection event that requires a function adjustment corresponding to a pointer position is generated by the user and causes the main control unit 102 to execute a process for adjusting the function when a selection event occurs . In a preferred embodiment, the selection event is set to detect a user bending forward the forefinger tip, i.e., a click on the air, as if pressing the virtual button ahead. 6, the fingertip coordinates are either lowered in height, moved toward the screen, or both occur at the moment when the user clicks in the air.

FIG. 7 shows an embodiment of a selection event detection process. In this embodiment, the selection event detection unit 110 receives the first reference point, that is, the finger end point data, in units of frames from the reference point pixel detection unit 104, and stores it in the RAM 74 (operation 200). On the other hand, the selection event detection unit 110 receives the image data from the first and second frame memories 122 and 124, extracts the outline of the hand, and stores the position of the feature point or the center point on the contour line in operation 202. The selected event detection unit 110 calculates a frame change amount between frames in the forward and backward directions (z-axis direction) and the vertical direction (y-axis direction) from the coordinates received from the reference point pixel detection unit 104, Is calculated (Step 204). The selected event detection unit 110 determines whether the cumulative coordinate change amounts DELTA Zf and DELTA Yf in the forward and backward directions (z-axis direction) or the vertical direction (y-axis direction) are larger than the reference values TH1 and TH2 ). If the cumulative coordinate change amounts DELTA Zf and DELTA Yf in the forward and backward directions (z-axis direction) and the vertical direction (y-axis direction) are both smaller than the respective reference values TH1 and TH2, The process returns to operation 200 (operation 214).

On the other hand, if it is determined in step 206 that the cumulative coordinate change amounts DELTA Zf and DELTA Yf in the forward and backward directions (z-axis direction) or up and down directions (y-axis direction) are larger than the reference values TH1 and TH2, (Step 208, step 210). The contour movement amount is calculated based on the contour movement amount. If it is determined in operation 210 that the movement amount of the outline is smaller than the reference value TH3, the selection event detection unit 110 regards that only the finger is clicked in a state where the hand does not move as a whole and transmits a selection event generation signal to the main control unit 100 (Operation 212). However, if it is determined in operation 210 that the movement amount of the outline is larger than the reference value TH3, the selection event detection unit 110 does not transmit the selection event generation signal because the hand has moved overall for pointing. This process is repeatedly performed unless a control command deactivation command is applied from the main control unit 100. [

Upon receiving the selection event detection signal from the selection event detection unit 110, the main control unit 100 instructs the operation control unit 112 to perform the function adjustment corresponding to the icon around the pointer. Accordingly, the operation control unit 112 controls the operation of the receiver elements such as the tuner 12 and the voice amplifier (not shown) so that the user can perform the intended function adjustment.

The function deactivation event detector 114 monitors whether a function deactivation event for terminating the control function according to the present invention is generated and transmits a function deactivation event generation signal to the main controller 102 when a function deactivation event occurs. An example of a function deactivation event is that there is no finger movement for a time of 5 seconds or more, for example.

The digital television receiver according to the present invention operates as follows.

8 shows the overall flow of the control function in the digital television receiver according to the present invention.

The user can conveniently adjust the function of the receiver by activating the control function according to the present invention by generating a predetermined event as a requirement for activating the function during watching the broadcast or using the Internet. Such a function activation event may be a gesture of a certain pattern, such as winking a single eye, bringing a palm or fist to a specific part of the face, pointing a finger or palm toward the monitor, (E.g., a time of 3 seconds or more).

As long as the power is on, the receiver continues to acquire the image in front of the receiver through the image sensors 54a and 54b, and monitors whether the above-mentioned function activation event occurs (operation 300). When the function activation event occurs, the system controller 60 determines whether the receiver is performing the broadcast reception mode or the Internet mode (operation 302). If the receiver is in the broadcast reception mode, the system controller 60 performs a broadcast reception control process (operation 304). On the other hand, if the receiver is performing the Internet mode, the system control unit 60 performs an Internet browsing control process (operation 306).

FIG. 9 shows a broadcast reception control process shown in FIG. 8 (Step 304).

The pointer display control unit 108 provides the OSD adjustment menu data to the video signal processing unit 40 and outputs a selection control signal so that the video signal processing unit 40 displays the video on which the OSD adjustment menu is superimposed on the broadcast video, ) (Step 320). In this state, the pointer positioning unit 106 estimates a position that the user intends to point to and determines a pointer position, and displays a pointer at the determined pointer position (operation 322).

The pointer positioning process will be described in more detail with reference to FIG.

The first and second image sensors 54a and 54b continuously acquire images and store them in the frame memories 122 and 126 (operation 340). The reference point pixel detection unit 104 detects the first reference point, that is, the pixels (155a and 155b in Fig. 5) corresponding to the user's forefinger end points in the image captured by the first and second image sensors 54a and 54b And determines the position of the corresponding pixel corresponding to the second reference point, that is, the center point between the weights. The pointer position determination unit 106 calculates the spatial coordinates of the first reference point Pf (Xf, Yf, Zf) and the second reference point Peye (Xeye, Yeye, Zeye) using the position data of the pixels (Step 342, Step 344). The pointer position determination unit 106 determines the coordinates of the pointer on the display screen 52 using the spatial coordinates of the first reference point Pf (Xf, Yf, Zf) and the second reference point Peye (Xeye, Yeye, Zeye) Ppoint (Xpoint, Ypoint, 0) is determined (operation 346). When the pointer coordinates are determined, the pointer display control unit 108 updates the OSD adjustment menu data in accordance with the new pointer coordinates and provides the OSD adjustment menu data to the video signal processing unit 40. [ Accordingly, the pointer is displayed at a position corresponding to the pointer coordinates updated on the screen, and the menu item highlight icon around the pointer is highlighted and emphasized in hue, brightness, or saturation (Step 348).

Referring again to FIG. 9, when a pointer is displayed at a position to be pointed in operation 322, the user selects an icon related to a point where the pointer is located by generating a selection event, that is, The adjustment function corresponding to this icon can be executed. The selection event detection unit 110 of the system control unit 60 outputs a selection event generation signal to the main control unit 100 when a selection event occurs (operation 324). The main control unit 100 allows the operation control unit 112 to control the voice amplifier or the tuner so that the corresponding adjustment function can be executed (operation 326). Accordingly, a function intended by the user can be easily implemented. On the other hand, if the user does not move his / her finger for a predetermined period of time while using the adjustment function or selects the 'OSD adjustment menu end' icon on the screen 328, the deactivation event detection unit 114 outputs a deactivation event detection signal to the main control unit 100 . In response to this signal, the main control unit 100 controls the pointer display control unit 108 so that the OSD adjustment menu is erased, and the reference point pixel detection unit 104, the pointer positioning unit 106, the pointer display control unit 108, And the selection event detection unit 110 to the standby mode.

Fig. 11 shows the Internet browsing control process shown in Fig. 8 (Step 306).

In the Internet TV mode, the main control unit 100 provides the Internet content data received via the network interface 80 to the video signal processing unit 40, thereby displaying the Internet content on the display 50 ). In this state, when the control function is activated, the pointer positioning unit 106 estimates a position that the user intends to point to and determines a pointer position, and displays a pointer at the determined pointer position (operation 362). The process of determining the pointer position is the same as described above with reference to FIG. When a pointer is displayed at a point to be pointed, the user can generate a selection event. The selection event detection unit 110 of the system control unit 60 outputs a selection event generation signal to the main control unit 100 when a selection event occurs (operation 364). The main control unit 100 extracts a hyperlink or a menu item at the current pointing position on the display 50 display face and requests the server related to the content corresponding to the extracted hyperlink or menu item ). In step 368, when the content is received via the network interface 80, the main control unit 100 causes the received content to be displayed on the display 50. [ On the other hand, if the user selects the 'exit OSD adjustment menu' icon on the screen (328), the OSD adjustment menu or the Internet TV mode itself is terminated (operation 370).

It will be understood by those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

For example, although the preferred embodiments of the present invention have been described above with reference to a digital television receiver, the present invention is not limited thereto and can be applied to other image display apparatuses as well. Such an apparatus includes, for example, an analog television receiver, a monitor for receiving and displaying a composite video signal or a video bit stream without the function of receiving a television signal, a projector for presentation, a PC, and other information devices. It is not. Further, the present invention is similarly applicable to other electronic apparatuses having a display unit and to which a user can apply operation commands.

In the above description, the coordinates (Xpoint, Ypoint, 0) of the pointer are determined by using the forefinger's end point Pf as the first reference point and the middle point Peye to the right and left eye center as the second reference point. In another embodiment of the present invention, the second reference point may be a body portion other than the center point between the eyes, for example, an eyeball center of one eye, a point on the shoulder contour, or an upper end of the head.

On the other hand, in the above description, as shown in Fig. 12A, a point (Ppoint) at which a virtual straight line connecting the end point Pf of the forefinger and the center of gravity Peye meets the display screen 52 is determined as a pointing point , A part of the screen is covered by the pointing hand, which may cause inconvenience to the user. 12B, the auxiliary reference point Pfup is assumed to be offset by an offset D1 which is more constant than the first reference point Pf, and the auxiliary reference point Pfup and the second reference point Pfup, (Ppoint2) at which a virtual straight line connecting the display surface (Peye) meets the display screen (52) as the pointing point. In another embodiment, the auxiliary reference point Peyedown is assumed to be below the offset D2 which is more constant than the second reference point Peye, and the first reference point Pf and the auxiliary reference point Peyedown are connected to each other The point Ppoint2 at which the virtual straight line meets the display screen 52 may be determined as the pointing point. 12D, after determining a point (Ppoint) at which a virtual straight line connecting the first reference point Pf and the second reference point Peye with the display display surface 52 meets, (Ppoint3) which is higher than the point (D3) as the pointing point. According to such modified embodiments, there is no problem that the screen is covered by the hand when the user is pointing, and there is an advantage that the pointing becomes easier because the user does not need to raise the hand.

As such, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a conceptual illustration of an example in which a user points to a point on a screen in a digital television receiver according to the present invention; Fig.

Figures 2a and 2b illustrate a coordinate system used to determine the coordinates of reference points in one embodiment.

3 is a block diagram of one embodiment of a digital television receiver according to the present invention;

FIG. 4 is a detailed block diagram of a program module executed in the system control unit shown in FIG. 3;

5 illustrates a process of determining a position of a first reference point or a second reference point in a control unit.

6 is a diagram showing a finger change at the moment the user clicks on the air.

7 is a flow chart showing an embodiment of a selection event detection process.

8 is a flow chart showing the overall flow of control functions in a digital television receiver according to the present invention;

9 is a flowchart showing a broadcast reception control process;

10 is a flow chart specifically illustrating pointer positioning and display steps in the process of FIG. 9;

11 is a flow chart showing an Internet browsing control process;

12A to 12D are diagrams for explaining examples for determining a pointing point on a display surface;

Claims (35)

A display unit having a display surface; A video signal processor for providing a video signal of a predetermined format to the display unit so that an image corresponding to the video signal is displayed on the display screen; An image capturing unit for capturing an image in front of the display surface; And A first reference point corresponding to a user's hand on the front of the display surface and a second reference point corresponding to the user's body using the image captured by the image capture unit, A control unit for estimating a pointing point by the user using an auxiliary reference point associated with each of the two reference points and controlling the video signal processing unit to display a predetermined pointer on the estimated pointing point on the display surface, The pointing point, A first auxiliary reference point located at a predetermined distance from the first reference point and a virtual straight line connecting the second reference point meet at a predetermined distance from the first reference point and the second reference point, And wherein a virtual straight line connecting the second auxiliary reference point located at a position where the second auxiliary reference point is located meets the display surface. The method according to claim 1, A broadcast signal receiving unit for receiving a broadcast signal, demodulating and decoding the received broadcast signal, and outputting a video bit stream; Respectively, Wherein the image signal processing unit converts the video bit stream into the format and outputs a format-converted video signal to the display unit so that an image corresponding to the format-converted video signal is displayed on the display surface. The system of claim 2, wherein the image capture unit Two or more image sensors disposed at different positions from each other in front of the image display device; And an image display device. 4. The apparatus of claim 3, wherein the image capture unit Two image sensors in which the horizontal position and the vertical position are arranged differently from each other; And an image display device. The method of claim 2, The video signal processor synthesizes the format-converted video signal with predetermined on-screen display (OSD) adjustment menu data, And an image corresponding to the format-converted video signal is fused to a predetermined on-screen display adjustment menu in the display unit, and is displayed together with the pointer. The method of claim 5, Wherein the control unit provides the OSD adjustment menu data to the video signal processing unit. The method of claim 6, Wherein the control unit provides the OSD control menu data to the video signal processing unit so that a menu item corresponding to the pointing point on the display unit is highlighted and displayed. The method of claim 5, And the image signal processing unit synthesizes modified OSD adjustment menu data to the format-converted video signal so that a menu item corresponding to the pointing point is highlighted and displayed on the display unit. The image processing method according to any one of claims 1 to 8, wherein the first reference point is an end point of one of the fingers, and the second reference point is a center of an eye of one eye An upper end of the head, and a point of the elbow. delete delete delete delete The method according to any one of claims 1 to 8, Selection event detection means for detecting a predetermined selection event from an image captured by the image capture unit; And the control unit controls the image display apparatus so that an operation corresponding to the pointing point is performed in response to the selection event. The method according to any one of claims 1 to 8, A function deactivation event detection means for detecting a predetermined function deactivation event from an image captured by the image capture unit; And the control unit clears the pointer on the display surface in response to the function deactivation event. A display unit having a display surface; An image capturing unit for capturing an image in front of the display surface; And A first reference point corresponding to a user's hand on the front of the display surface and a second reference point corresponding to the user's body using the image captured by the image capture unit, A controller for estimating a pointing point by the user on the display surface using an auxiliary reference point associated with each of the two reference points and displaying a predetermined pointer on the estimated pointing point on the display surface, The pointing point, A first auxiliary reference point located at a predetermined distance from the first reference point and a virtual straight line connecting the second reference point meet at a predetermined distance from the first reference point and the second reference point, And a virtual straight line connecting a second auxiliary reference point located at a point where the first auxiliary reference point intersects with the display surface. 17. The image capture device of claim 16, And two or more image sensors disposed at different positions on the front surface of the electronic device. 18. The apparatus of claim 17, And two image sensors arranged such that the horizontal position and the vertical position are different from each other. delete delete delete delete The method according to any one of claims 16 to 18, Selection event detection means for detecting a predetermined selection event from an image captured by the image capture unit; Wherein the control unit controls the electronic device to perform an operation corresponding to the pointing point in response to the selection event. The method according to any one of claims 16 to 18, A function deactivation event detection means for detecting a predetermined function deactivation event from an image captured by the image capture unit; And the control unit clears the pointer on the display surface in response to the function deactivation event. 1. An electronic device comprising a display screen and an image sensor, the electronic device comprising: (a) imaging the front of the display surface using the image sensor; (b) determining a position of each of the first reference point corresponding to the user's hand and the second reference point corresponding to the user's body in the captured image; (c) estimating a pointing point by the user on the display surface using an auxiliary reference point related to the position of each of the first and second reference points; And (d) displaying a predetermined pointer at the estimated pointing point, The pointing point, A first auxiliary reference point located at a predetermined distance from the first reference point and a virtual straight line connecting the second reference point meet at a predetermined distance from the first reference point and the second reference point, And a virtual straight line connecting a second auxiliary reference point located at a position where the second auxiliary reference point is located and the display plane. [Claim 25] The method of claim 25, wherein the first reference point is an end point of one of the fingers, the second reference point is a center of an eyeball, a center point of the eyeball, a point on a shoulder contour associated with the finger, Wherein the point is selected from a group of singularities constituted by points. delete delete delete delete 26. The method of claim 25 or claim 26, (e) detecting a predetermined selection event by the image sensor, and causing an operation corresponding to the pointing point to be performed in response to the selection event; Further comprising the steps of: 26. The method of claim 25 or claim 26, Monitoring whether a predetermined function deactivation event occurs; And When the function deactivation event occurs, erasing the pointer on the display surface; Further comprising the steps of: 33. The method of claim 32, Wherein the pointing point is outside the display surface for a predetermined period of time or longer. A display unit having a display surface; A video signal processor for providing a video signal of a predetermined format to the display unit so that an image corresponding to the video signal is displayed on the display screen; An image capturing unit for capturing an image in front of the display surface; And A first reference point corresponding to a user's hand located in front of the display surface and a second reference point corresponding to the user body are determined using the image captured by the image capture unit, And a point located at a predetermined distance from a point at which a virtual straight line connecting the position of the second reference point with the display surface meets the pointing point by the user, And a controller for controlling the video signal processor to display a predetermined pointer on the estimated pointing point on the display surface. An operation control method for an electronic device having a display display surface and an image sensor, the operation control method allowing a user to operate the electronic device without contacting the electronic device, Imaging the front of the display surface using the image sensor; Determining a first reference point corresponding to the user's hand and a second reference point corresponding to the user's body in the captured image; Estimating, as a pointing point by the user, a point located at a predetermined distance from a point at which a virtual straight line connecting the position of the first reference point and the position of the second reference point meet the display surface; And And displaying a predetermined pointer at the estimated pointing point.

Images