JP2010003303A - Controller for electronic equipment - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a controller for electronic equipment for achieving both flexibility to various electronic equipment and the convenience of a remote operation and for eliminating the need to use equipment for a remote operation such as a remote controller at hand. <P>SOLUTION: The controller for electronic equipment 1 images a target object, displays an image for control for controlling electronic equipment on a display device 23, generates a detection signal showing a change in an imaged image obtained by imaging the target object when the target object performs a prescribed operation to the image for control, calculates the degree that the cycle of a signal waveform of the detection signal coincides with a predetermined cycle, and decides that the electronic equipment is controlled when the degree of coincidence becomes equal to or greater than the predetermined value. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a control device for an electronic device, and more particularly to a control device for remotely operating an electronic device having a display device such as a television receiver or a personal computer.

  In the 1980s, infrared remote controllers (commonly known as remote controllers) came to be attached to household appliances such as television receivers, and user interfaces that could be controlled at hand became widespread, greatly changing the usage of household appliances. At present, this type of operation is still mainstream, but the remote control is based on a mechanism that executes one function with one push. For example, in a television receiver, “power”, “channel”, “volume”, “input switching”. Such a key corresponds to this, and it has been a very convenient remote operation method for a conventional television receiver.

  However, recently started data broadcasting is complicated and difficult to use because the “up / down / left / right” and “decision” keys of the remote controller are pressed many times in order to select a desired menu screen. Further, EPG (Electronic Program Guide) has a similar problem because it selects a desired position from a guide screen arranged in a matrix and presses a key.

  In Patent Document 1, in order to solve such a problem, position designation information obtained by a mouse or a position designation operation device similar thereto is encoded into a key depression time series code which is a time series pattern of a key depression signal. A control device has been proposed in which the key press time-series code is transmitted to a television receiver.

JP 2003-283866 A JP 2002-196855 A JP-A-6-153017 JP 2001-307107 A JP 2002-149302 A JP-A-8-212327 JP 2000-075991 A JP 2000-196914 A JP-A-8-044490 JP-A-9-081307 JP 2001-282456 A

  The control device disclosed in Patent Document 1 remotely operates a television receiver by a pointing operation very similar to that of a personal computer (personal computer). Therefore, it is difficult for those who do not use a personal computer, and it is impossible to introduce usability of a personal computer as it is from the viewpoint of information literacy (ability to use information). There is a need for a new operation means that matches the usage pattern of today's television receivers that require remote operation.

  Furthermore, the progress of networking will accept information by displaying various information obtained from storage media in the house and the Internet outside the house on the display of a television receiver or a personal computer. Since the operation form in this case depends on the information source, it is necessary to cope with various operation forms, and the current remote control attached to the home electric appliance cannot cope with it.

  As described above, the conventional remote controller enlarges the remote controller with respect to the diversification and complication of functions of the television receiver and the like. When the selection operation of the menu screen such as data broadcasting is performed with a conventional remote controller, the role as a pointing device becomes more important, so the conventional remote controller has many drawbacks in terms of usability. In addition, networking has a problem that the functions of all devices connected to the network are controlled via a display device, and the number of remote controllers increases in proportion to the number of connected devices. This problem is a phenomenon that is often experienced in the present situation because it is impossible to determine which remote controller is actually the corresponding remote controller even in a VTR, video disc, or other audio device connected to the television receiver. . Furthermore, in the Internet, it is obvious that information on all websites is selected and controlled, and it is no longer in the form of the conventional remote control.

  The present invention has been made paying attention to the above-described points, and achieves both flexibility and convenience of remote operation for various electronic devices, and does not require the use of a remote operation device such as a remote control at hand. An object of the present invention is to provide a control device for electronic equipment.

The present invention provides the following electronic device control method in order to solve the above-described problems of the prior art.
In the control method for controlling the electronic device 1 having the display device 23, an object is imaged, a control image for controlling the electronic device is displayed on the display device, and the target object is displayed with respect to the control image. Generates a detection signal indicating a change in a captured image obtained by capturing the target object when performing a predetermined operation, obtains a degree of coincidence of a period of a signal waveform of the detection signal with a predetermined period, and A control method for an electronic device, comprising: confirming that the electronic device is to be controlled when a degree to be performed is equal to or greater than a predetermined value.

  According to the present invention, the mirror image conversion image signal obtained by the operator shooting with the video camera and subjected to mirror image conversion and the operation image signal including the operation button are mixed, that is, superimposed and displayed on the display device. Control operation of the electronic device corresponding to the operation image when it is detected that the operator displayed on the screen in this state points to the operation image, and further that the operator has operated the operation image. Is executed. More specifically, an area of the operator's hand indicating the operation area corresponding to the operation button in the mirror image conversion image signal is extracted, and data indicating a change in the image signal of the hand area in the image signal of the operation area is detected. Is done. Then, a waveform based on the detected data is generated, and when it is detected that the operator has operated the operation image based on the degree of coincidence obtained by comparing this waveform with data of a preset waveform. Then, the control operation of the electronic device corresponding to the operation image is executed. Therefore, it is possible to remotely control the electronic device without using a remote control device such as a remote controller. In this case, the operation image can be appropriately set according to the operation to be selected or the content of information, so that both flexibility for various electronic devices and convenience of remote operation can be achieved.

It is a figure for demonstrating the outline | summary of the electronic device operating method of this invention. It is a block diagram which shows the structure of the principal part of the electronic device (television receiver) concerning one Embodiment of this invention. It is a figure for demonstrating the method of the mirror image conversion of an image. It is a figure which shows an operator's image and the image for operation. It is a figure for demonstrating the state on which the operator's image and the image for operation were superimposed (mixed). It is a figure for demonstrating a response | compatibility with the detector shown in FIG. 2, and the detection area on the screen of a display apparatus. It is a block diagram which shows the structure of the detector shown in FIG. It is a block diagram which shows the structure of the object extractor shown in FIG. It is a figure for demonstrating the hue and saturation of the target object which are extracted with an object extractor. It is a flowchart of the process which calculates a hue from a color difference signal. It is a figure which shows the luminance signal level of the target object extracted with an object extractor. It is a figure which shows the state which held the hand on the operation button on the screen. It is a figure which shows the state which bent the finger | toe on the operation button on a screen. It is a time chart which shows the change of the projection area on the screen of a hand in case the operation | movement which bends a finger twice is performed. It is a figure which shows the state with which the operation button and the image of the hand overlapped. It is a figure which shows the frequency distribution and average luminance level (APL) of the brightness in the time T-1 shown in FIG. It is a figure which shows the frequency distribution and average luminance level (APL) of the brightness in the time T0 shown in FIG. It is a figure which shows the frequency distribution and average luminance level (APL) of the brightness in the time T1 shown in FIG. It is a figure which shows the frequency distribution and average luminance level (APL) of the brightness in the time T2 shown in FIG. It is a figure which shows the frequency distribution of brightness and the average luminance level (APL) in the time T3 shown in FIG. It is a figure which shows the frequency distribution of brightness and the average luminance level (APL) in the time T4 shown in FIG. It is a figure which shows the frequency distribution and average luminance level (APL) of the brightness in the time T5 shown in FIG. It is a figure which shows the frequency distribution and average luminance level (APL) of the brightness in the time T6 shown in FIG. It is a functional block diagram for demonstrating operation | movement of the control information judgment device (CPU) shown in FIG. It is a time chart for demonstrating the data processing for determining operation | movement which selects an operation button. It is a block diagram which shows the structure of the digital filter for removing a DC offset component. It is a figure for demonstrating the specific structure of the digital filter shown in FIG. 26, and the relationship with a processing waveform. It is a time chart which shows transition of the cross correlation calculation value of operation detected and predetermined selection operation. It is a figure for demonstrating the process when the image of the operator image | photographed with the video camera is too small. It is a figure for demonstrating the process when the image of the operator image | photographed with the video camera is too large. It is a figure which shows the apparatus for changing a video camera and its angle of view.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram for explaining a difference between an operation mode of a conventional remote controller and an operation mode of the present invention. When the viewer (user) 3 operates the television receiver 1, conventionally, the viewer 3 holds the remote control device 4 in his hand and presses a key for performing a desired function toward the television receiver 1. Operate by. Therefore, if the remote control device 4 is not provided, the operation cannot be performed and the user is sometimes inconvenienced.

  On the other hand, in the present embodiment, the television receiver 1 is provided with the video camera 2, the viewer 3 is photographed by the video camera 2, and the operation of the viewer 3 is obtained from the image of the video camera 2, The television receiver 1 and related devices are operated. The operation of the viewer 3 is specifically the movement of the body (hand, foot, face, etc.) that selects a desired button from the menu screen displayed on the television receiver 1. Further, when the surroundings of the television receiver 1 are dark, the same effect can be obtained by using something other than the body, for example, a pen light. In the present embodiment, the most realistic hand movement method is employed.

  FIG. 2 is a block diagram showing a configuration of the television receiver 1. The television receiver 1 includes a reference synchronization generator 11, a timing pulse generator 12, a graphics generator 16, a video camera 2, and a mirror image converter. 14, a scaler 15, a first mixer 17, a pixel number converter 21, a second mixer 22, a display device 23, a detection unit 19, and a control information determination device (CPU) 20.

  The reference synchronization generator 11 generates a horizontal period pulse and a vertical period pulse that serve as a reference for the television receiver 1. When a video signal is input when a television broadcast is received or from an external device, a pulse that is synchronized with the synchronization signal of the input signal is generated. The timing pulse generator 12 generates a pulse having an arbitrary phase and width in the horizontal and vertical directions required for each block shown in the figure. As shown in FIG. 1, the video camera 2 is positioned in front of the television receiver 1 and shoots an image in front of the viewer (user) 3 or the television receiver. The output signal of the video camera 2 is a luminance (Y) signal and a color difference (R−Y, B−Y) signal, and is synchronized with the horizontal period pulse and the vertical period pulse output from the reference synchronization generator 11. In the present embodiment, it is assumed that the number of pixels of the image captured by the video camera 2 matches the number of pixels of the display device 23. If the number of pixels does not match, a pixel number converter may be inserted.

  The mirror image converter 14 is for displaying the subject image photographed by the video camera 2 on the display device 23 with the left and right reversed like a mirror. Therefore, when displaying characters, the left and right are reversed in the same way as a mirror. As shown in FIG. 3, this can be easily realized by performing an address operation using the first memory and the second memory for writing and reading data in the horizontal period. For example, when one horizontal period is N pixels, the input signal is alternately written for each horizontal period from the write addresses 0 to N-1 of the two memories, the read operation is performed exclusively with the write operation, and N is opposite to the write address. Trace from -1 to 0 and flip the image in the horizontal direction left and right. In this embodiment, a method of inverting the image in the horizontal direction using the memory is adopted. However, when a CRT (Cathode Ray Tube) is used as the display device 23, the horizontal deflection is operated in reverse. Similar effects can be obtained. In that case, it is necessary to reverse the graphics and other images on the mixing side in the horizontal direction in advance.

  The scaler 15 adjusts the size of the subject image photographed by the video camera 2, and adjusts the enlargement ratio and the reduction ratio two-dimensionally under the control of the control information determination unit (CPU) 20. Also, horizontal and vertical phase adjustments can be performed without performing enlargement / reduction. How to use this will be described in detail later.

  The graphics generator 16 develops the menu screen transferred from the control information determination unit (CPU) 20, and the signals on the memory are R (red) signal, G (green) signal, B (blue) signal. Even if the primary color signal is developed, the output signal synthesized or mixed with the video signal in the subsequent stage is a Y (luminance) signal and a color difference signal (RY, BY). The number of graphics planes to be generated is not limited, but only one plane is necessary for explanation. In this embodiment, the number of pixels is made to match the number of pixels of the display device 23. If they do not match, it is necessary to add a pixel number converter to match.

The first mixer 17 mixes the output signal Gs of the graphics generator 16 and the output signal S1 of the scaler 15 by controlling the mixing ratio with the control value α1. Specifically, the output signal M1o is expressed by the following equation.
M1o = α1 · S1 + (1−α1) · Gs
The control value α1 is set to a value between “0” and “1”. When the control value α1 is increased, the proportion of the scaler output signal S1 increases and the proportion of the graphics generator output signal Gs decreases. The example of the mixer is not limited to this, but in the present embodiment, the same effect can be obtained as long as two types of input signal information are included.

  The detection unit 19 includes a first detector 31, a second detector 32, a third detector 33, a fourth detector 34,. The number of detectors included in the detection unit 19 does not specify the number, but is a push button that represents the control contents linked with the menu screen generated by the graphics generator 16 (corresponds to an operation button that is clicked with a mouse on a personal computer). It fluctuates corresponding to the number of.

  The control information determination unit (CPU) 20 analyzes data output from the detection unit 19 and outputs various control signals. The processing content of the control information judging device (CPU) 20 is realized by software, and the algorithm will be described in detail. In the present embodiment, processing by hardware (each functional block) and software (development on the CPU) are mixed, but the present invention is not particularly limited to the separation shown here.

  The pixel number converter 21 performs pixel number conversion for matching the number of pixels of the external input signal input from the outside with the number of pixels of the display device 23. The external input signal is assumed to be a signal input from the outside of the television receiver such as a broadcast TV signal (including data broadcast) and a video (VTR) input. Although the synchronization system of the external input signal is omitted from the description here, the synchronization signal (horizontal and vertical) is acquired, and the reference synchronization generator 11 matches the synchronization.

The second mixer 22 has the same function as the first mixer 17. That is, the output signal M1o of the first mixer 17 and the output signal S2 of the pixel number converter 21 are mixed by controlling the mixing ratio with the control value α2. Specifically, the output signal M2o is expressed by the following equation.
M2o = α2 · M1o + (1-α2) · S2
The control value α2 is set to a value between “0” and “1”. When the control value α2 is increased, the ratio of the first mixer output signal M1o is increased and the ratio of the pixel number converter output signal S2 is increased. Get smaller. The example of the mixer is not limited to this, but in the present embodiment, the same effect can be obtained as long as two types of input signal information are included.

  The display device 23 is assumed to be a CRT (cathode ray tube), an LCD (liquid crystal display), a PDP (plasma display), a projection display, or the like, but the display method of the display is not limited. The input signals of the display device 23 are a luminance signal Y and color difference signals RY and BY, and are converted into RGB primary color signals and displayed inside the display device 23.

  The operation of the television receiver 1 configured as described above will be described along with the operation of the viewer (user). FIG. 4 shows a graphics image 41 and a scaler output image 43 in which the video camera image is mirror-image transformed and the number of pixels is matched with that of the graphics image 41 by scaling, and corresponds to the two images 41 and 43. The signal is mixed in the first mixer 17. The graphics image 41 forms a menu screen for performing control, and a square 42 represents a push button (operation button). One scaler output image 43 is a photograph of the user and displayed in the same manner as a mirror. A broken-line square 44 in the scaler output image 43 represents a detection area of the detection unit 19 including a plurality of detectors 31 and 32 and is arranged at the same position as the push button 42 on the graphics image 41. Has been.

  FIG. 5 is an image diagram for explaining the mixing process in the first mixer 17. FIG. 5A shows a control menu screen generated by the graphics generator 16 and includes push buttons (1-1) to (1-8). FIG. 5B is a screen (mirror image conversion and scaler processed) that captures the user with the video camera 2, and detection areas (2-1) to (2-8) corresponding to the detectors of the detection unit 19. It is included. FIG. 5C shows a screen obtained by mixing the screens shown in FIGS. 5A and 5B at the mixing ratio corresponding to the control value α1 in the first mixer 17. Note that the user image displayed in FIG. 5C actually has a lower brightness and contrast than the image shown in FIG. 5B corresponding to the control value α1.

  In this way, the user's image is displayed on the display device 23 in the same manner as a mirror, and the control menu screen is also displayed superimposed. Thereby, the user himself / herself can recognize his / her movement and the position of the control menu. The control is performed by moving the hand (body) so that the user looks at the screen and touches the push button on the screen. In other words, control is performed by the user operating while looking at the screen so that the hand is positioned on the push button. If the push button touched (applicable) and the hand in the detection area 1: 1 are detected, it can be recognized that the corresponding push button has been pressed. Therefore, a control signal corresponding to the control content of the corresponding push button can be output from the control information determination device (CPU) 20. At that time, the push button on the graphic menu screen changes its shape and / or color to indicate on the display that the operation has been recognized. For example, the image is changed as if the push button was pressed.

  FIG. 6 shows the correspondence between the detection areas (2-1) to (2-8) set on the image from the video camera 2 and the detectors 31 to 34 of the detection unit 19, and further shows the detection area. Timing pulses (horizontal and vertical) that specify (2-1) and (2-8) are shown.

  As shown in FIG. 7, each of the detectors 31 to 34 includes an object extractor 51, a timing gate device 52, and an object feature data detector 53. The timing gate unit 52 controls the passage of the image signal from the video camera 2 with the timing pulse shown in FIG. The passing area is in the detection area indicated by a broken-line rectangle in FIG. The user's hand captured by the video camera 2 is extracted by further filtering the signal limited within the detection area.

  The object extractor 51 includes a filter according to the characteristics of the image. In the present embodiment, in order to detect the user's hand, the object extractor 51 performs a filter process that pays particular attention to the skin color. Specifically, as shown in FIG. 8, the object extractor 51 includes a specific color filter 71, a gradation limiter 72, a combiner 73, and an object gate 74. The specific color filter 71 will be described with reference to FIG. FIG. 9 is a color difference plan view, in which the vertical axis is RY and the horizontal axis is BY. All color signals of a television signal can be represented by vectors on this coordinate and can be evaluated in polar coordinates. The specific color filter 71 limits the hue and color density (saturation degree) of the color signal input as the color difference signal. In order to specify this, the hue is expressed by a counterclockwise angle with the BY axis in the first quadrant as a reference (zero degree). The saturation is a vector scalar quantity, indicating that the origin of the color difference plane is zero and no color is present, and the saturation increases as the distance from the origin increases and the color is dark.

  In FIG. 9, the range extracted by the specific color filter 71 is set to an angle θ2 that is smaller than the angle θ1 corresponding to the equal hue line L1 and corresponding to the equal hue line L2, and the color density is equal saturation. It is set in a range smaller than the degree line S2 and larger than S1. This range of the second quadrant corresponds to a skin color region which is the color of the human hand extracted in the present embodiment, but is not particularly limited thereto. The specific color filter 71 detects whether or not the color difference signals (R−Y, B−Y) coming from the video camera 2 are in an area surrounded by the equal hue line and the equal saturation line. For this purpose, the angle and the saturation are calculated from the color difference signal and determined.

  In the angle calculation, for example, an angle formed on the color difference plane shown in FIG. 9 is calculated for each input pixel by a process as shown in FIG. FIG. 10 illustrates the angle calculation process in a flowchart, but the angle calculation process may be realized by either software or hardware. In the present embodiment, it is realized by hardware. In step S401 in FIG. 10, it is detected from the sign of the color difference signal RY, BY component of each input pixel that the hue of the input pixel is located in the first quadrant on the color difference plane. In step S402, the larger one of the absolute values of the color difference signals RY and BY components is calculated as A, and the smaller one is calculated as B.

In step S403, the angle T1 is detected from B / A. This angle T1 is 0 to 45 °, as is apparent from the processing in step S402. The angle T1 can be calculated by broken line approximation or a ROM table. In step S404, it is determined whether A is | R−Y |, that is, whether or not | R−Y |> | B−Y |. If not | R−Y |> | B−Y |, the process directly proceeds to step S406. If | R−Y |> | B−Y |, the angle T1 is replaced with (90−T1) in step S405. Thus, tan ⁻¹ ((R−Y) / (B−Y)) is obtained.

The angle T1 detected in step S403 is set to 0 to 45 ° because the tan ⁻¹ ((RY) / (BY)) curve suddenly increases when the angle exceeds 45 °. This is because it is unsuitable for calculation.
Further, at step S406, it is determined whether or not the second quadrant is used by using the quadrant data detected at step S401. If it is the second quadrant, T = 180−T1 is calculated at step S407. If it is not the second quadrant, it is determined in step S408 whether it is the third quadrant, and if it is the third quadrant, T = 180 + T1 is calculated in step S409. If it is not the third quadrant, it is determined in step S410 whether or not it is the fourth quadrant, and if it is the fourth quadrant, T = 360−T1 is calculated in step S411. If it is not the fourth quadrant, that is, the first quadrant, the angle T is set to T1 (step S412). Finally, in step S413, the angle T formed on the color difference plane of FIG. 9 for each input pixel is output.

  Through the above processing, the angle of the input color difference signals RY and BY on the color difference plane can be obtained in the range of 0 to 360 °. Steps S404 to S411 are processes for correcting the angle T1 detected in step S403 to the angle T. In steps S404 to S411, the angle T1 is corrected according to the first to fourth quadrants.

Next, the saturation, which is the color density, is calculated by the following equation.
Vc = sqrt (Cr × Cr + Cb × Cb)
However, Cr is the (RY) axis component of the color signal as shown in FIG. 9, and Cb is the (BY) axis component. Sqrt () is an operator that performs a square root operation.

The processing here does not specify software or hardware, but multiplication and square root are not easy to implement in hardware, and software is not preferable because of many operation steps, so it can be approximated as follows: .
Vc = max (| Cr |, | Cb |)
+ 0.4 × min (| Cr |, | Cb |)
However, max (| Cr |, | Cb |) is an arithmetic process for selecting the larger one of | Cr | and | Cb |, and min (| Cr |, | Cb |) is | Cr | This is the process of selecting the smaller of | Cb |. Further, Vc is a vector scalar quantity, and here represents saturation.

  Whether or not the angle (hue) T and the degree of saturation Vc obtained from the above are within the range of the angle of the uniform hue line θ1 to θ2, and the color density is smaller than the equal saturation line S2 and larger than S1. evaluate. It is the role of the specific color filter 71 shown in FIG. 8 to pass signals within the range.

  As shown in FIG. 11, the gradation limiter 72 of FIG. 8 limits a specific gradation of the luminance signal. The 8-bit digital signal has 256 gradations from 0 to 255. A maximum level Lmax and a minimum level Lmin that limit gradations in this range are set, and a luminance signal with the level in between is output.

  The synthesizer 73 converts the signals input from the specific color filter 71 and the gradation limiter 72 into region pulses. That is, when both the signal that has passed through the specific color filter 71 and the signal that has passed through the gradation limiter 72 exist (AND), a high level pulse is output.

  The region pulse generated by the synthesizer 73 is supplied to the object gate 74. The object gate 74 passes the luminance signal and the color difference signal when the region pulse is at a high level. When the range outside the region pulse (when the region pulse is at a low level), that is, when the input signal is not allowed to pass, a signal having a specified value is output. In the present embodiment, a black level luminance signal and a saturation zero color difference signal are output within a range that does not pass.

  The function of the object extractor 51 is to limit the color signal hue (angle) and saturation level of the specific color filter 71 with the luminance signal level. By limiting the hue and saturation with the specific color filter 71, it can be narrowed down to the human skin color, but the human skin changes depending on the sunburn, and also varies depending on the race. Although these skin colors are various, human hands can generally be detected by adjusting the hue, saturation, and the gradation-limited range of the luminance signal. In this embodiment, human hands are targeted. However, if other hues and gradations are appropriately adjusted, they can be distinguished from others in the same manner in the parameter change category in this embodiment.

  FIGS. 12A and 13A show that a push button area (broken line square) is set for an image input from the video camera 2. In the present invention, a hand is held over a video camera image, and the operation is discriminated as control information. 12B and 13B show the movement of the hand. FIG. 12B shows the hand held over the push button, and FIG. 13B shows a state where the finger of the hand is bent. In the present embodiment, the operation of bending the finger is repeated twice to recognize that the push button has been pressed. It goes without saying that the same effect can be obtained even if the hand is held up or the operation is performed once or three times or more.

  Next, with reference to FIG. 14, the operation of the hand will be described along the time axis. The elapsed time from the state where the finger is extended to the state where the finger is bent is defined as Tm. In order to make the explanation easy to understand, the time is divided from T0, T1,..., T6 with this as the minimum unit of operation. The following will be described along with the passage of time.

  At time T-1, a state where no hand is present is shown, at time T0, a state where the hand is held over a desired push button is shown, and from time T1, the hand is held for a minimum Tm period, and at time T2, The hand finger is folded, and at time T3, the hand finger is folded from the folded state, and at time T4, the hand finger is folded again. At time T5, a state in which the finger of the folded hand is extended again is shown. From time T6, the state in which the finger of the hand is extended is held for a minimum Tm period, and at time T7, a case where no hand is present is shown. Yes.

  The above is the movement of the hand on the push button. The waveform diagram on the upper side of FIG. 14 shows changes in the projected area of the palm on the push button that is modeled and drawn on the lower side. By analyzing this waveform and identifying the movement of the hand, the control action can be captured.

  FIG. 15 shows an image corresponding to the output signal of the object extractor 51. The signal passing through the object extractor 51 is limited to the push button area by the timing gate, and the other areas are ignored. In the timing gate, the color is specified by the specific color filter 71, where only the skin color region of the hand is passed, and also limited to the gradation direction, only the limited region is passed, and the other regions are black ( Both the luminance signal and the color signal are zero). Therefore, as shown in FIG. 15, the fingers other than the fingers of the hand are black (shown with hatching). FIG. 15A shows a state where a finger is held over, and FIG. 15B shows a state where the finger is bent. Here, the reason why the area other than the skin color area is set to black is to increase the detection accuracy of the subsequent detector, and the optimum level (specific level) is set according to the object to be detected.

  The feature of the image is further detected from FIG. The object feature data detection unit 53 shown in FIG. 7 is a functional block that detects various features from an image, that is, a histogram detector 61, an average luminance (APL) detector 62, a high frequency detector 63, and a minimum value detector. 64 and a maximum value detector 65. There are other items that characterize the image, but in this embodiment, it is determined that the user is a hand based on these contents, and the action is recognized.

  These detectors 61 to 65 are configured by hardware in this embodiment, and process the state in the detection frame in the space into data in a screen unit (field and frame unit: vertical cycle unit) to control information. The data is sent to the bus of the judgment device (CPU) 20. A control information determination unit (CPU) 20 stores the data in a variable and performs data processing.

  The histogram detector 61 divides the gradation of the luminance signal into 8 steps and counts the number of pixels present in each step, and determines the frequency value of each step for each screen (field or frame screen). (CPU) 20 to output. Similarly, the APL detector 62 adds the luminance levels in one screen, and outputs the average luminance value of one screen divided by the number of pixels to the control information determination unit (CPU) 20. The high frequency detector 63 extracts a high frequency component by a spatial filter (two-dimensional filter), and outputs the frequency of the high frequency component to the control information determination unit (CPU) 20 in units of one screen. The minimum value detector 64 outputs the minimum gradation value of the luminance signal in one screen, and the maximum value detector 65 outputs the maximum gradation value of the luminance signal in one screen to the control information determination unit (CPU) 20.

  The control information determination unit (CPU) 20 stores the received data in a variable and performs software processing. From this point onward, software processing is performed in the present embodiment.

  16 to 21 are diagrams showing output data of the histogram detector 61 and the average luminance detector 62. FIG. This indicates a gradation histogram and average luminance (APL), and the APL is indicated by an arrow so that the size can be sensibly understood. In these figures, the frequency is shown in the vertical axis, and the gradation (brightness) is shown in eight steps on the horizontal axis. Case 1 and case 2 have different hand brightness, and case 1 is a case where the frequency of the hand is concentrated on a specific step, and case 2 represents a case where the hand straddles two steps. 16 to 21 are histograms corresponding to the times T-1 and T0 to T6 shown in FIG. Since the object extractor 51 has a specific skin color and the gradation is limited and the other portions are black, the frequency on the histogram is concentrated on the black portion and the gradation limited portion.

Hereinafter, the transition of the histogram and the value of APL will be described with time.
FIG. 16, time T-1: When there is no hand, there is basically no frequency, but it represents the frequency of a component that is in the skin tone and in the limited gradation area that exists in the background captured by the camera. . APL is low.

  FIG. 17, time T0: In the state where the hand is held over the desired push button, Case 1 represents that the frequency of the gradation in Step 5 increases. In case 2, since it extends over the regions of steps 5 and 6, the frequency does not increase as much as in case 1, but the total amount of steps 5 and 6 increases by almost the same amount. APL is proportional to the hand area because it increases.

FIG. 18, time T1: When the hand is held for a minimum Tm period, it is the same as time T0.
FIG. 19, time T2: With the finger of the hand folded, the black area expands and the frequency of step 5 decreases. APL also decreases.
FIG. 20, time T3: The state in which the finger of the hand is extended after being bent is the same as that at time T0.
FIG. 21, time T4: Same as time T2, with the finger of the hand bent again.
FIG. 22, time T5: The same as time T0, with the finger of the folded hand extended again.
FIG. 23, time T6: The state in which the hand is stretched out is held for the minimum Tm period, and is the same as at time T0.

  In the histogram data processing, since the portion that has not been extracted as an object in this embodiment is black, Step 1 is a black region and is excluded from the evaluation target. There are various ways of evaluation. In this embodiment, since the information of the direct hand is steps 4, 5, 6, and 7 that fall within the range in which the gradation is limited, these are summed to evaluate the value.

  The fluctuation of the total value of the histogram is similar to the fluctuation of APL, and the hand motion can be determined by the same analysis only by changing the value scale. In order to increase the accuracy of discrimination, evaluation is performed using different detection means. The detection data of the high frequency detector 63, the minimum value detector 64, the maximum value detector 65 and the like described above are evaluated in the same manner, and the accuracy can be improved by discriminating from more different detection data. In this embodiment, the high frequency detection detects the frequency of the edge component between the finger and the black background, and as can be seen from FIG. 15, the edge component decreases when the finger is bent, and the fluctuation is a histogram. And similar to APL. However, in the maximum value detection, if any part of the finger is in any part of the push button, there is no significant change and the detection accuracy is poor. In addition, since the portion outside the object extraction range is black this time, the minimum value detection does not change, so it does not make any sense. However, depending on the color and gradation of the object captured by the video camera, it may be better to make the portion out of the object extraction range white. The accuracy of determination is increased by applying a color or white having a contrast to the target object to a portion outside the object extraction range.

  The evaluation method for discrimination is to discriminate the movement of a predetermined hand, the basics of the algorithm are the same, and the scale differs depending on the detection item. The output of the object feature data detection unit 53 shown in FIG. 7 is transferred to the control information determination unit (CPU) 20 via the CPU bus. The control information judging device (CPU) 20 detects an operation and outputs an appropriate control signal. FIG. 24 is a functional block diagram for explaining the operation of the control information determination unit (CPU) 20. The control information determination unit (CPU) 20 includes a first operation detector 201 corresponding to the histogram detector 61, A second motion detector 202 corresponding to the average luminance (APL) detector 62, a third motion detector 203 corresponding to the high frequency detector 63, and a fourth motion detector corresponding to the minimum value detector 64 204, a fifth motion detector 205 corresponding to the maximum value detector 65, and a control information generator 210. The functions of the motion detectors 201 to 205 and the control information generator 210 are actually realized by software processing.

  The motion detectors 201 to 205 output detection flags according to the detection results of the corresponding detectors 61 to 65 and supply the detection flags to the control information generator 210. The control information generator 210 evaluates the contents of the input detection flag and outputs appropriate control information. The evaluation here is performed using an operator in a computer language, such as logical sum, logical product, exclusive logical sum,>, <, etc., and an arithmetic expression including these operators is satisfied. A control signal is output. In this embodiment, the logical sum of the outputs of all the motion detectors 201 to 205, that is, control information is output when at least one motion detector outputs a detection flag.

  The motion detectors 201 to 205 will be described in detail. Here, the operation is detected from the total data of the limited steps of the histogram. FIG. 25 shows the fluctuation of data obtained from the histogram. Times T0 to T6 shown in this figure correspond to times T0 to T6 shown in FIG. FIG. 25A shows the fluctuation of the total value of 4 to 6 steps limited in the histogram. It corresponds to the part other than black and represents the area occupied by the hand. FIG. 5B is obtained by removing the offset component. If there is no change during the Tm period, the value becomes zero. FIG. 6C is the result of squaring the data of FIG. 5B, and the finger bending action is close to a sine wave, and the square is a sine wave having a period of ½. FIG. 4D is an integration of the waveform of FIG. 2A, and shows a gate pulse for discriminating hand movements with arrows. FIG. 6E shows the correlation with hand movement by cross-correlating the waveform of FIG. Furthermore, the final judgment, that is, the process of guiding the user to operate the push button is shown over (D) and (E) in FIG.

FIG. 25 will be described in detail below.
(A) is the same as the waveform shown with the hand movement in FIG. In particular, portions that do not contribute to discrimination are omitted. Here, let Tm be the period from when the hand is stretched to when it is bent, and consider it as the reference period (time interval) for hand movement. The period from time T0 to T1 is a period in which the hand is held, and here, it is determined that the corresponding push button is activated for about Tm period. Time T1 to T2 is a process of bending the finger of the hand. Times T2 to T3 represent the process of extending the finger of the hand. Time T3 to T4 is a process of bending the finger of the hand again. Time T4 to T5 is a process of extending the finger of the hand again. At time T5 to T6, the finger is stretched and held there for about Tm period. This series of actions is similar to a mouse double-click and can be operated without difficulty for humans.

  (B) is obtained by removing the DC offset component. A configuration example of the filter to be removed is shown in FIG. This filter includes a delay unit 81 that delays the input signal by time Tm, a subtracter 82 that subtracts the input signal from the output signal of the delay unit 81, and an attenuator 83 that attenuates the signal level by half. This is a high-pass digital filter that requires a tap coefficient of (−1/2, 1/2) before and after the Tm period. The delay in the Tm period is for passing the hand movement waveform. The form of the digital filter is not particularly limited to this, but here, the simplest configuration and a small absolute delay amount are adopted. When the sign of the tap coefficient is changed to (1/2, -1/2), the output waveform is inverted, but this does not affect the subsequent processing, and either is used. It will be effective.

  (C) is the square of the waveform of (B) as described above, and the period is halved when considered as a sinusoidal waveform. FIG. 27 shows details of the waveform of (C) obtained by correlation calculation with a digital filter, which is shown in detail in FIG. 27A, 27B, and 27C are the same as FIG. What is drawn below is a cross-correlation digital filter. What is described as D in the square represents a delay of one vertical cycle, and corresponds to a D flip-flop (hereinafter referred to as DFF) used in a digital circuit. K0, k1,..., K8 represent tap coefficients of the digital filter. Σ means a sum operation of tap coefficient outputs, and a cross-correlation calculation value Y (nτ) is obtained at the output. The waveform of the hand motion that takes the cross-correlation is the result of processing (C), and one waveform exists as the value of the tap coefficient, and the shape of the waveform is completely due to the cross-correlation between the hand motion and the tap coefficient. When the values match or are close to each other, the cross-correlation digital filter output becomes the maximum value.

The arithmetic processing by the digital filter shown in FIG. 27 can be expressed by the following mathematical formula. In the configuration example of FIG. 27, N is “9”.

  The digital filter shown in FIG. 27 receives a waveform drawn in (C) from the right side. The reason that the signal is input from the right is because the waveforms shown in (C) coincide with each other (partly arrows) when they enter the DFF. Therefore the last hand movement comes to the far right side without delay. The coefficient of cross correlation may be included in the waveform (C) as it is. However, scale is not mentioned here. In addition, since it is not preferable that the coefficient value is biased in terms of DC, the DC offset should be prevented as much as possible with reference to the center of the waveform level of (C). Since the coefficient value is used to calculate the correlation with the motion of the hand, if the content of the motion defined here is changed, the coefficient value is changed accordingly, and the control device can be controlled with a different motion. You can also run multiple digital filters with several correlation values, each with a different meaning. Since there are 9 tap coefficients here, there are 9 vertical periods. In the case of a video signal having 60 vertical periods per second, it is about 1/7 second, and the hand movement is a little faster, but in order to slow it down, it can be slowed and adjusted by increasing the tap coefficient. The optimal number of tap coefficients may be set by experiment.

  Here, the waveform of (C) is considered as the correlation count value, but the waveform from time T1 to T5 of the waveform shown in (B) may be used. However, the accuracy is higher when the waveform (C) is used.

  Returning to FIG. 25, the waveform (E) will be described. The waveform (E) is obtained by modeling the value appearing at the output when the waveform (C) is input to the cross-correlation digital filter. As the time advances, the degree of coincidence of the hand movement with the tap coefficient increases and the peak increases. The evaluation of this waveform is converted into an absolute value as indicated by an arrow, integrated further (envelope in the figure), and it is determined that it is a hand action to be controlled when the threshold value TH4 is exceeded. . The absolute value here is the same even if it is integrated after squaring as shown in FIG.

  Here, the size of the waveform shown in (C) is based on the first histogram data, and the accuracy of determination varies depending on this size. Also, initialization (reset) is required for integration, that is, cumulative addition. The waveform shown in (D) fulfills this. Furthermore, the process which suppresses the meaningless information for raising detection accuracy is required.

  The waveform (D) is obtained by integrating the data from the histogram (A) along the time axis. The period up to time T1 is a period in which the hand is held up, and (B) is zero when (A) does not change, that is, only the high frequency component is picked up. This means that it is still when you hold your hand. Naturally, it is a human motion, but there are fluctuations. However, if it is a large motion, it can be regarded as still, and the processing can be cut off by setting a dead zone that can be easily realized. Therefore, when (B) is between the threshold values TH1a and TH1b and (A) is equal to or greater than the threshold value TH0, integration is performed and the integrated value increases. The waveform is (D). In the initial state where the integration is started, initialization is performed when the waveform of (A) is less than or equal to the threshold value TH0. When the integral value shown in (D) exceeds the threshold value TH2, it is activated. The area so far is called a detection area.

  A period having a specific time constant is set as indicated by an arrow Y1 after the integral value shown in (D) exceeds the threshold value TH2, and the period until the subsequent determination area is entered is called an active area. The active region is a region for which correlation calculation is performed. The determination area is an area where the hand movement is determined as the control state based on the result of the correlation calculation, and starts when the cross-correlation digital filter output is converted into an absolute value and integrated exceeds the threshold value TH4. Another threshold TH3 is provided for the waveform of (D). The threshold value TH3 is a value smaller than the threshold value TH2, and the period indicated by the arrow Y2-1 exceeding the threshold value TH3 is used for the determination. The bending operation of the finger of the hand is finished by extending the last finger. The threshold value TH3 is for confirming the last operation. A period indicated by an arrow Y2-2, which is an area opposite to the arrow Y2-1, is considered as a period during which the determination is prohibited, and misrecognition is avoided. When the arrow Y1, the arrow Y2-1, and the arrow Y3 are aligned, it is determined that the hand movement is control information in the determination area.

  This detection area is an area for recognizing that the hand is held over, and when the area is exceeded, the color and shape of the graphics control image can be changed to actively guide the user. Similarly, when the active area is completed and when it is determined in the determination area, it is possible to change the color and shape of the graphics control image to make the user recognize it. This has the advantage of not requiring the user to try again.

  Next, the absolute level of the threshold value TH3 will be described. The size of the output value of the cross-correlation digital filter varies depending on the range of the area where the hand is held. In particular, even if it is fixed, it is only necessary for the person to hold his / her hand over the necessary area, but it is also necessary to absorb the ambiguity of the person. Here, the time Tr required for the integration region up to the threshold value TH2 is seen and adjusted accordingly. In the period Tr, the larger the area where the hand is held, the shorter the period, and conversely, the time becomes longer when the area is small. Therefore, when the time Tr is large, the threshold values TH3 and TH4 are increased, and when the time Tr is small, the threshold values TH3 and TH4 are corrected to be small. This adaptive control suppresses the dependence on the hand-held area and enables accurate recognition.

  In the operation method of the present embodiment, it is conceivable that the hand size is small or the push button position is not reached depending on the position of the person with respect to the angle of view of the video camera. In this case, it is necessary to adjust the “enlargement”, “reduction”, and “phase (position)” of the image with the scaler 15 shown in FIG. FIG. 29 is a diagram for explaining this. The example shown in FIG. 29A is a screen that is small and difficult to control. In this case, the scaler 15 is controlled to “enlarge”, and the “phase” is adjusted so that the person is located at the center of the screen, so that an appropriate size and an appropriate position as shown in FIG. Users can be displayed.

FIG. 30 shows an example in which the scaler 15 is controlled to “reduce” and the “phase” is adjusted so that a person comes to the center of the screen.
As another method, as shown in FIG. 31, the video camera 2 is attached to the camera platform, and the user's image is displayed in an appropriate size and position by controlling the angle of view by the camera platform and zooming control of the video camera 2. You may make it make it.

  The operation method of the present invention is to operate while looking at the mirror image superimposed on the control image developed on the graphics displayed on the display device, but it may be difficult to see because the images are superimposed. . In that case, the mixing ratio of the first mixer 17 or the second mixer 22 is controlled with the timing pulse (horizontal and vertical) for specifying the region on the detector screen shown in FIG. To do. For example, in order to make it easy to see the push button as the control image, the first mixer 17 may be controlled with a timing pulse so as to reduce the value of the control value α1. By finely controlling the mixer as described above, the relationship between the viewer and the control image (push button) can be easily understood.

In the embodiment of the present invention described above, the hardware and software are described separately. However, the method of the separation may be any way.
The operation method of the present embodiment is an operation in which a push button drawn on a symmetrical graphic is touched using a human hand. All the operations that the conventional push buttons have, such as “Power”, “Channel”, “Volume”, “Input switch”, “Up”, “Down”, “Left”, “Right”, “Determine”, etc. It is possible to replace it.

  By using the operation method as described above, it is possible to control a control target without using a control device such as a remote controller. Therefore, troubles related to the remote control such as battery exhaustion and contact failure are eliminated. In addition, when there are a plurality of remote control devices, it has been necessary to search for a remote control device to be controlled. However, in the operation method of the present invention, it is sufficient if there is a human body, and such a problem is solved.

  In addition, since the control content is displayed on the screen, it is possible to handle various control devices. This is important in a networked environment, and controls a network device through a device having a display device, in other words, by transferring a control screen of a device connected to the network to a device having a display device through the network. Since this screen becomes the control screen, any control screen can be supported.

  Also, to control from the menu screen with the remote control, it is necessary to operate while repeatedly viewing the hand and the screen because the operation of selecting the push button on the menu screen while pressing the button on the hand many times . On the other hand, according to the operation method of the present invention, the user himself / herself overlaps on the operation screen so that the user can always see and operate the screen, not only can the operation be performed quickly, but the eyes are hard to be tired. Can be improved.

  In addition, hand movement can be similar to that of a personal computer mouse. In this embodiment, the operation of bending the finger of the hand in a double click manner is adopted. Personal computers are widespread in modern times, and being close to the operation of personal computers is easy for many users to handle and is preferable in terms of information literacy.

The present invention is not limited to the embodiment described above, and various modifications can be made. For example, the mixing ratios of the first mixer 17 and the second mixer 22 shown in FIG. 2 may be effectively combined in the level direction, the space direction, and the time direction. As a result, various combinations of images from the video camera 2 and the generated graphics menu screen and broadcast program video are possible, and various information can be exchanged at the same time.
The electronic device to be operated according to the present invention is not limited to a television receiver, but a personal computer, a combination of a television receiver and a video disk recorder, or a combination of a television receiver and a video tape recorder. Any device having a display device may be used.

1 Television receiver (electronic equipment)
2 Video camera 4 User (operator)
12 Timing pulse generator 14 Mirror image converter (mirror image conversion means)
15 Scaler 16 Graphics Generator 17 First Mixer (First Mixing Means)
19 Detection unit (detection means)
20 Control information judging device (CPU) (control means)
22 Second mixer (second mixing means)
23 display device 51 timing gate device (position determination means)
52 Object extractor (object identification means)
53 Object feature data detection unit (change detection means)

Claims (1)

In a control method for controlling an electronic device having a display device,
Image the object,
Displaying a control image for controlling the electronic device on the display device;
Generating a detection signal indicating a change in a captured image obtained by capturing the target object when the target object performs a predetermined operation on the control image;
Obtain the degree to which the period of the signal waveform of the detection signal matches a predetermined period,
A method for controlling an electronic device, comprising: determining that the electronic device is to be controlled when the degree of coincidence is equal to or greater than a predetermined value.

Images