JP2010079332A - Remote operation device and remote operation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote operation device with excellent visibility and operability, which operates an apparatus to be controlled without touching the display surface of the remote operation device with a finger, and to provide a remote operation method thereof. <P>SOLUTION: The remote operation device includes: an operation display means 117 for displaying the operation button or menu for remotely operating the apparatus to be controlled; an imaging means 116 for imaging an operator to perform the operation, based on the displaying of the operation display means 117; a hand area detecting means; a hand operation detecting means; and a control means. The hand area detecting means detects the area of the hand from an image photographed by the imaging means 116. The hand operation detecting means detects the operation of the operator, based on the shape and movement of the hand area detected by the hand area detecting means and the displaying of the operation display means. The control means transmits the signal of the operation detected by the hand operation detecting means to the apparatus to be controlled through the use of a communication means. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a remote operation device and a remote operation method for remotely operating an electronic device.

  Conventionally, when an electronic device such as a television or a DVD recorder is remotely operated, an operation of pressing a button fixedly provided on the remote controller is generally used. However, since electronic devices are complex and highly functional, remote controllers for remotely operating such devices increase the number of operation buttons, making it difficult to find buttons for desired operations, or as desired The operation of the remote controller has become very troublesome, such as increasing the number of times the button is pressed to perform the operation. Therefore, the operation is difficult not only for the elderly and the information weak, but also for people who have many opportunities to come into contact with the above-mentioned devices on a daily basis. In view of this, a remote controller has been proposed in which a display device and a touch panel are provided in the remote controller, and only necessary buttons are displayed on the liquid crystal touch panel by changing the display contents (see, for example, Patent Document 1). .

JP 2005-136518 A (pages 9 to 11, FIGS. 4 to 7 and 9)

    In the conventional remote operation system, only the necessary buttons are displayed on the LCD touch panel, and the buttons are pressed. However, since the display surface is directly touched with a finger, it becomes difficult to see and operates as visibility decreases. There was a problem that the sex deteriorated.

  The present invention has been made to solve the above-described problems, and operates a control target device without touching the display surface of the remote operation device with a finger, so that the remote operation device has good visibility and operability. And a remote control method.

  A remote operation device according to the present invention includes an operation display unit that displays an operation button or menu for remote operation of a control target device, an imaging unit that images an operator who operates based on the display of the operation display unit, and a hand region A detection means, a manual operation detection means, and a control means are provided. The hand region detection means detects the hand region from the image taken by the imaging means. The manual operation detection means detects an operation by the operator based on the shape and movement of the hand area detected by the hand area detection means and the display of the operation display means. The control means transmits an operation signal detected by the manual operation detection means to the control target device using the communication means.

  According to the present invention, the device to be controlled is operated based on the shape and movement of the area of the operator's hand that is operated based on the display of the operation display means. Therefore, a remote control device with good operability can be obtained.

Embodiment 1 FIG.
FIG. 1 is a device configuration diagram of a remote operation system including a display device 112, a recording / playback device 113 as a control target device, and a remote operation device 111 according to Embodiment 1 of the present invention. A video / audio input of a display device 112 such as a television and a video / audio output of a recording / playback device 113 such as a DVD recorder are connected by a connection cable 114 such as an AV (Audio / Visual) cable for transmitting video / audio. A remote control device 111 such as a remote controller communicates with the display device 112 and the recording / playback device 113 by infrared or wireless transmission signals 115 and remotely controls them. The remote operation device 111 includes, for example, an imaging unit 116, a liquid crystal touch panel unit 117, and hardware buttons 118. The cradle 119 may be rechargeable.

  FIG. 2 is a block diagram constituting remote operation device 111 according to Embodiment 1 of the present invention. The operation display unit 121 displays virtual operation buttons, menus, and the like for remote operation of control target devices such as the display device 112 and the recording / playback device 113 so that the operator can understand the operation status. The imaging unit 122 captures an operator's hand using a CCD (Charge Coupled Device) camera or the like, and the hand region detection unit 123 detects a hand region from the image captured by the imaging unit 122. The manual operation detection unit 124 detects the operation of the operator from the shape and movement of the hand region detected by the hand region detection unit 123. The control unit 126 instructs the operation display unit 121 to display the operation status and operable button display based on information such as the detection result of the manual operation detection unit 124, and communicates with the main device through the communication unit 125.

A specific configuration example for configuring this functional block diagram is shown in FIG. In FIG.
A CPU (Central Processing Unit) 131 performs various processes using a memory 132 such as a flash ROM (Read only memory) storing program codes and a RAM (Random access memory). The liquid crystal panel 133 displays virtual operation buttons and menus, and the touch panel 134 detects that a virtual operation button and menu displayed on the liquid crystal panel 133 are pressed. The liquid crystal panel 133 and the touch panel 134 are integrated to form a liquid crystal touch panel 135. The signal transmission / reception unit 136 transmits / receives a transmission signal 115 corresponding when the operator performs an input operation on the liquid crystal touch panel 135. The sound generation device 137 includes a speaker or the like, and generates an operation sound with a beep sound or the like when the operator performs an input operation on the liquid crystal touch panel 135. The imaging unit 138 is configured with a CCD camera or the like, and images an operator's hand and the like.

  The touch panel 134 is arranged on a liquid crystal panel 133 that displays virtual operation buttons and menus, and the CPU 131 detects coordinates when the operator presses the touch panel 134 with a finger or a pen. Thus, the virtual operation button or menu displayed on the liquid crystal panel 133 can be detected.

  A basic operation of the remote operation device 111 configured as described above will be described. The CPU 131 built in the remote operation device 111 displays virtual operation buttons and menus on the liquid crystal touch panel 135 in accordance with the program code stored in the memory 132. FIG. 4 shows a liquid crystal display unit of the remote operation device 111, and shows an example of three display modes of the remote operation device 111 and virtual operation buttons displayed on the liquid crystal touch panel 135.

  FIG. 4A shows a television mode in which virtual operation buttons are arranged mainly when operating the display device 112, and FIG. 4B shows a virtual operation mainly when operating the playback function of the recording / playback device 113. FIG. 4C shows a reservation mode in which virtual operation buttons are mainly arranged when the recording reservation function of the recording / reproduction device 113 is operated. Each of these modes is realized by the CPU 131 appropriately changing the shape / size / position / color / character of the virtual operation button displayed on the liquid crystal touch panel 135. In each mode, virtual operation buttons (television mode button 141, reservation mode button 142, and playback mode button 143) for transition to the display mode are arranged in common.

  Next, the operation in each display mode will be described. In the television mode shown in FIG. 4A, a virtual operation button (“broadcast switching” button in the figure) for switching broadcast types such as terrestrial broadcasting and satellite broadcasting is displayed on the liquid crystal touch panel 135. When the virtual operation button is selected by the operator, the broadcast type of the display device 112 such as a television is switched. Further, virtual operation buttons (1 to 12 in the figure) corresponding to selectable reception channels are displayed on the liquid crystal touch panel 135, and when the virtual operation button corresponding to the reception channel is selected by the operator, a television or the like is displayed. The receiving channel of the display device 112 is switched.

  In the playback mode shown in FIG. 4B, virtual operation buttons (“HDD” and “DVD” in the figure) corresponding to the names of reproducible recording media provided in the recording / playback apparatus 113 such as a DVD recorder are displayed on the liquid crystal touch panel. 135. It also displays virtual operation buttons for fast-forwarding, fast-rewinding, and stopping playback images, and virtual operation buttons equivalent to the up and down keys for selecting programs to be played from a recorded program list called a playback list. To do.

  In the reservation mode shown in FIG. 4C, virtual operation buttons for operating the electronic program guide are displayed on the liquid crystal touch panel 135. This is for operating a program table by an EPG (Electrical Program Guide) provided in a display device 112 such as a television or a recording / playback device 113 such as a DVD recorder. In addition, a virtual operation button that enables a change operation of the time / reception channel to change the reservation content is also displayed.

  As described above, in each mode, virtual operation buttons necessary for “viewing a TV program”, “playing a recorded program”, and “program recording reservation”, which are intended by the operator, are arranged.

  Next, transition of each display mode will be described. 5A to 5C show the remote operation device 111 (FIG. 5A) and the display device 112 when the television mode is changed to the reproduction mode among the three display modes of the remote operation device 111. (FIG. 5C) and the operation of the recording / reproducing apparatus 113 (FIG. 5B) are shown. When the remote operation device 111 is in the television mode (FIG. 4A), when the playback mode button 143 is selected, the operation screen of the remote operation device 111 is changed to the playback mode (FIG. 4B).

  At this time, the remote control device 111 transmits a transmission signal 115 for displaying the external input video to the display device 112 at the same time, and prepares to display the output video / audio of the recording / playback device 113. At the same time, the transmission signal 115 for displaying the reproduction list is transmitted to the recording / reproducing apparatus 113 to prepare for outputting the reproduction list.

  FIG. 5D illustrates an example of the transmission signal 115. In order to operate the display device 112, a television identification code and an input switching code are transmitted, and then the recording / reproducing device 113 is operated. Recorder identification code and playlist display code are sent.

  Through these series of operations, the video / audio of the playlist displayed by the recording / playback device 113 is displayed on the display device 112 via the connection cable 114. As described above, the playback mode has virtual operation buttons necessary to select a program from the playlist, so that the operator can select a recorded program to be played immediately after switching to the playback mode. Thus, the usability of the remote control system can be improved.

  6 (a) to 6 (c) show a remote control device 111 (FIG. 6 (a)), a display device 112 (FIG. 6 (c)) and a recording / reproducing device 113 (FIG. 6) when the television mode is changed to the reservation mode. 6 (b)) is shown. When the reservation mode button 142 is selected, the mode changes to the reservation mode.

  FIG. 6 (d) shows an example of the transmission signal 115. In order to operate the display device 112, a television identification code and an input switching code are transmitted, and subsequently, to operate the recording / reproducing device 113, Recorder identification code and program guide display code are transmitted.

  As described above, the display mode transition from the TV mode to the playback mode and the display mode transition from the TV mode to the reservation mode are given as examples, but the liquid crystal touch panel is used in the other modes and menu layers as well. Thus, only buttons that can be operated according to the state of the device at that time can be displayed, and the device can be remotely operated according to the selected button, so that the usability can be improved.

  Now, in general, there are various types of touch panels, such as a resistive film type, and although the sensitivity characteristics differ somewhat depending on the method, there is a problem that fingerprints are attached to the touch panel in order to detect that the touch panel is pressed. is there. In addition, when charging is possible with the cradle 119 as shown in FIG. 1, the operability is improved if the cradle 119 can be operated while it is placed on the cradle without being bothered and held.

  Therefore, the operation of the remote operation device 111 that captures an image of the operator's hand by the imaging unit 122 and detects and operates the gesture (shape, position, etc.) will be described. In order to actually perform a gesture operation, an on / off operation of the gesture operation function is also required. This on / off operation may be performed with the hardware button 118, but in the case of a remote control capable of two-way communication with the main body, the gesture operation of the remote control may be turned on / off depending on the operating state of the main body.

  FIG. 7 shows an example of button operation by hand gesture. The upper part of the figure shows the hand imaged by the imaging unit 138 and the center point (white circle) of the palm at that time, and the lower part shows the display of the liquid crystal touch panel 135. Here, the focus (button selection) is moved by moving the position while holding the hand by detecting whether the hand is open, grasping, or raising the finger, and the center position of the palm. The button is pushed by raising the finger. That is, a selection operation is detected from the shape of the hand, and a movement operation is detected from the movement of the hand shape and the center point of the hand.

  The focus does not move from the button “2” even if the hand is moved as shown in FIG. When the hand is held as shown in FIG. 7B, the focus can be moved. For example, when the hand is moved in parallel as shown in FIG. 7C, the movement amount of the center position is detected and the focus position is changed. For example, “2” moves to “program pause” and “playback”. For example, when the user wants to press the “play” button, when the focus position is “play”, the finger is raised as shown in FIG. 6D and the hand is held again at that position to release the button. “Play” button processing is executed, and the display on the liquid crystal is updated as shown in FIG. 7E. After that, when the hand is opened as shown in FIG. 7F, the focus does not move even if the hand is moved. The button operation is complete. If the button operation that is in focus when the finger is raised is cancelled, the opening operation can be performed without grasping the hand, or the hand can be moved as it is. It should be noted that it is desirable to change the display of the highlight display indicating the focus position depending on the state of the hand (whether the hand is opened, grasped, or the finger is raised).

  FIG. 8 shows the flow of this gesture detection process. The hand region detecting step S220 in the hand region detecting means 123 which is the center of the gesture detecting process is composed of steps 221 to 224, and the hand operation detecting step S230 in the hand operation detecting means 124 is in steps 231 to 233. Composed. The hand region detection unit 123 includes a binarization correction unit, a distance calculation unit, a center emphasis correction unit, and a hand region determination unit. The hand operation detection unit 124 includes a contour direction calculation unit, a center contour distance calculation unit, and the like. Consists of manual operation determination means. These means are constituted by, for example, the CPU 131 and the memory 132 in the configuration diagram of FIG. In each means, the program code stored in the memory 132 is calculated by the CPU 131, so that the hand region detection step S220, the binarization correction step S221, the distance calculation step S222, the center weight correction step S223, the hand An area determination step S224, a manual operation detection step S230, a contour direction calculation step S231, a center contour distance calculation step S232, and a manual operation determination step S233 are executed.

  A hand gesture detection method will be described with reference to the flowchart of FIG. 8 and the explanatory diagrams of FIGS. After the control means 126 performs an operation display step S200 in which an operation button or menu for remote operation of the device to be controlled is displayed on the operation display means 121, the imaging means 122 is in the vicinity of the remote operation device 111 and is displayed as an operation display means. Imaging of the operator who operates based on the display of 121 is started, and imaging step S210 for acquiring an image in a predetermined time unit is executed. When the operator reaches for the remote control, the captured image acquired in this step is, for example, an image as shown in FIG.

  As shown in FIG. 8, the hand region detection step S220 includes a binarization correction step S221, a distance calculation step S222, a center weight correction step S223, and a hand region determination step S224. Each step is executed by the corresponding means as described above.

  In the binarization correction step S221, an area based on a predetermined signal is first converted into an HSV conversion (Hue, Saturation, Value: hue, saturation, brightness) color space in order to reduce the influence of brightness. For example, the skin color area is extracted as binarized information. In this binarization, a hole area or a protrusion area in the skin color area may be generated as noise, and these are corrected by an area expansion process or a contraction process. For example, by increasing the color range to be detected and increasing the process of filling the hole area by the expansion process, it is possible to reduce the noise and obtain a binarized image as shown in FIG.

  In general, when binarization processing is performed based on color information, if a similar color area exists in the area corresponding to the background, this part is also extracted, resulting in noise. In such a case, a background image memory may be prepared to obtain a background image at all times, and a condition that a difference from the background image is an area having a predetermined signal value or more may be set. In addition, as described later, since the ease of use is determined from the shape, it is possible to cope with some misdetection at the binarization stage. Although the case where skin color is extracted by HSV conversion has been described here, binarization may be performed only by luminance information, or binarization may be performed by detecting a hand region using boosting used in face detection. good. As described above, based on a predetermined signal from the captured image, a skin color region, a difference region from the background image, and the like are extracted, and noise processing is performed to obtain a binarized image.

  In the distance calculation step S222, the distance from the point inside each region to the shortest contour in the binarized image region obtained in the binarization correction step S221 is calculated, and the maximum distance point at which the distance to this contour is the maximum is calculated. This position is detected and set as the center candidate of the hand in each region. In FIG. 9 (c), a plurality of closed dashed lines in the extraction region are equidistant lines having the same distance to the contour, and the (+) point in each region is the center candidate of the hand. Here, the reason why the center of gravity (x) of the area is not set as the center of the hand is that the arm area is affected by the arm area when the arm area is formed with clothes without sleeves as shown in FIG. This is because there is a deviation from the center.

  In the center emphasis correction step S223, a correction value is added to the maximum distance point value in each area calculated in the distance calculation step S222 so that the center of the captured image becomes maximum at the end of the image and becomes lower. For example, this correction value may be conical or hemispherical with the center of the image as the apex. In this way, it is highly likely that the target hand is photographed at the center of the image, and disturbances such as a face are present at the edge of the image. This is because there is a high possibility that an object will be photographed.

  For example, as shown in FIG. 9C, a hand and face area having the same size is detected, and the value of the maximum distance point of the hand area is, for example, “10”, and the value of the maximum distance point of the face area is For example, in the case of “12”, the face may be determined as a hand candidate if it is simply determined by the output of the distance calculation step S222. On the other hand, since the maximum distance point of the hand region is close to the center of the screen, the correction value of the center-weighted addition value is “9”, for example, and the maximum distance point of the face region is close to the edge of the screen, By adding these as “3”, the value of the hand area is “19” and the value of the face area is “15”, so that the hand area close to the center can be reliably detected. Although the correction values are added here, it is of course possible to combine multiplications and other calculations, or the correction values may be used.

  In the hand region determination step S224, there is a high possibility that the region including the point with the largest distance maximum point value corrected in the center emphasis correction step S223 is the hand region, and the distance maximum point value is the largest. It is determined that the point is likely to be the center point of the hand region, and handness is determined for this region. For example, as shown in FIG. 9D, the area above the center point of the hand of the area is determined as a shape determination area, and the area, contour length, height, width, and the like of the area are calculated. When these are within a predetermined range, it is determined that the region is a hand region. For example, when the width is extremely short and only 1/5 of the height, it is determined that the region is not a hand region. In this way, the hand region and the center point are determined, and when it is determined that the region is not the hand region, the screen displayed on the operation display unit 121 is left as it is (step S200), and the process returns to the imaging step S210 and waits for image acquisition. Become. If a hand region is detected, the process continues to the next manual operation detection step S230.

  As shown in FIG. 8, the manual operation detection step S230 includes a contour direction calculation step S231, a center contour distance calculation step S232, and a manual operation determination step S233. Each step is executed by the corresponding means as described above.

  In the contour direction calculation step S231, for example, the region shown in FIG. 9D is tracked in units of pixels along the contour indicated by the thick line as the effective region of the hand, for example, shown in the direction code of FIG. Thus, for example, from 0 to “7”, the contour direction is counterclockwise such that “0” is the upper right direction from the center position, “1” is the upper direction, and “2” is the upper left direction. Are encoded in 8 directions. In FIG. 10A, when the contour tracking start point when the hand is held is (S) and the end point is (E), the direction code near the start point (S) is “1” in the upward direction. It can be seen that the direction code changes almost monotonously until the direction code near the point (E) becomes the downward direction “5”.

  On the other hand, in FIG. 10 (c), when the contour tracking start point when one finger is raised is (S) and the end point is (E), the finger of the raised finger is located near the contour tracking start point (S). The direction code is “1” at a location corresponding to the position, and the direction code is “5” at locations other than the vicinity of the end point (E). Also, in FIG. 10D, when the contour tracking start point when the hand is opened is (S) and the end point is (E), the opposite direction codes (in this case, the directions “1” and “5”). And 4 repetitions of “2” and “6”).

  In the center contour distance calculating step S232, for example, the region shown in FIG. 9D is tracked in units of pixels along the contour indicated by a thick line as an effective region of the hand in the same manner as the contour direction calculating step 231, for example, FIG. The distance from the center of the hand region indicated by the (+) position is calculated. For example, when holding the hand as shown in FIG. 11A, the center contour distance from the contour tracking start point (S) to the end point (E) is a small value as a whole, and the maximum value of the center contour distance is It does not exceed twice the minimum value of the center contour distance (dotted line).

  On the other hand, the center contour distance from the contour tracking start point (S) to the end point (E) in the case of standing one finger as shown in FIG. 11 (a) is the maximum value at the fingertip position, and this maximum value is the center value. The value is larger than twice the minimum value of the contour distance (dotted line). Further, the center contour distance from the contour tracking start point (S) to the end point (E) when the hand is opened as shown in FIG. 11C is a maximum value corresponding to the number of open fingers. These local maximum values are larger than twice the minimum value of the center contour distance (dotted line).

  In the manual operation determination step S233, using the direction code and the center contour distance information obtained in the contour direction calculation step S231 and the center contour distance calculation step S232, for example, whether the hand is opened or held, or a finger is raised. Judge whether it is. As a specific example, when contour tracking is performed and the change in the direction code from the start point (S) to the end point (E) is substantially uniform and the maximum value of the center contour distance is smaller than twice the minimum value, When there is a pair of opposite direction codes such as “holding a hand”, direction codes “1” and “5” and the maximum value of the center contour distance is larger than twice the minimum value, When there are multiple sets of opposite-direction codes such as “standing”, direction codes “1” and “5”, and multiple maximum values of the center contour distance are greater than twice the minimum value, “open your hand” It is determined. That is, the selection operation is detected from the shape of the hand region, and the movement operation is detected from the shape of the hand region and the movement of the center point of the hand region. Note that either one or both of the results obtained in the contour direction calculating step S231 and the center contour distance calculating step S232 may be used. When either one is used, the calculation process can be performed quickly, and when both are used, the detection accuracy of manual operation can be increased.

  Here, when it is determined that the operation is not a manual operation because the manual operation cannot be detected, the screen displayed on the operation display unit 121 is left as it is (step S200), the process returns to the imaging step S210, and the image acquisition is waited.

  When it is determined that the operation is a manual operation, the operation content by the operator is detected based on a display position such as a virtual operation button or a menu highlight displayed on the operation display unit 121. That is, as in the operation of FIG. 7, for example, depending on the focus position of the button when the hand moves and the position of the hand moves, or the position of recognizing the operation of raising the finger again, which virtual operation button or menu is selected. Detect whether the operation content. Thereafter, the control unit 126 in the remote operation device 111 executes a communication step S240 for transmitting an operation signal to the control target device through the communication unit 125. Further, the control means 126 performs an operation display step S200 for displaying by the operation display means such as highlighting the operation button or menu display according to the detected operation or operation status. Thereafter, the process returns to the imaging step S210, waits for image acquisition, and repeats until the gesture operation ends. The gesture operation may be ended by using the hardware button 118 or may be ended by performing a specific operation.

  In the remote operation device and the remote operation method configured as described above, the hand of the operator who uses the camera provided in the remote operation device 111 and operates based on the display of the operation display means 121 in the vicinity. Controlled devices such as TVs and recorders are operated from the shape and movement, so that the display surface of the remote control device is not touched and soiled, so visibility is not lowered and a remote control device with good operability is obtained. It is done. In addition, convenience can be improved because it can be operated with only one hand without having a remote control.

  In addition, when the camera is mounted on a control target device such as a television and the control target device is operated by determining the shape and movement of the operator's hand, the operator can move to an arbitrary position. The relative positional relationship between the operator and the operator is infinitely different, and the operation position cannot be narrowed down. As a result, the size of the hand in the photographed image changes variously, and the accuracy of detecting the hand movement deteriorates. However, by mounting the camera on the remote control, the operator approaches the remote control for remote control operation, so that the assumed positional relationship between the operator and the remote control can be limited. Therefore, the position and size of the hand in the photographed image can be limited, the recognition rate of the hand motion is improved, the operation is easily determined, and the operability is improved.

Embodiment 2. FIG.
In the first embodiment, the remote operation device 111 is used as a commander by a simple button operation, but here, a case will be described in which the operation status is displayed on a television or the like to be used like a mouse in a personal computer.

  FIG. 12 is an equipment configuration diagram of a remote operation system including the display device 112, the recording / reproducing device 113, and the remote operation device 111 according to Embodiment 2 of the present invention. The configuration is basically the same as in FIG. 1, the functional block diagram of the remote control device is the same as in FIG. 2, the hardware configuration is the same as in FIG. 3, and the gesture detection process is also from FIG. 8 to FIG. The same.

  In the description of the first embodiment using FIG. 7, an example has been described in which the hand gesture operation status is displayed on the liquid crystal display of the remote operation device 111 itself, and the focus position movement and button operation are performed. However, in FIG. 12, the remote operation device 111 is set to the mouse operation mode, and a signal for displaying the operation and hand movement from the remote operation device 111 on the control target device is transmitted to the display device 112 and the recording / reproduction device 113 as the transmission signal 115 By transmitting, the movement of the hand is displayed on the display device 112, and the mouse operation on the screen of the browser or the like displayed on the display device 112 is enabled.

  Specifically, a menu or operation button is displayed on the operation display means 121 to enter the mouse operation mode (operation display step S200). Further, in this way, in the step of processing based on the display of the operation display means 121, unlike the first embodiment, the display content of the remote operation device 111 is not updated. Thereafter, a gesture operation is performed in the same manner as in the first embodiment to acquire an image (step S210), a hand region is detected (step S220), and a hand operation is detected (step S230). The detected operation and a signal for displaying this operation on the device to be controlled are transmitted as the transmission signal 115 to the display device 112 and the recording / reproducing device 113 using the communication unit 125 (communication step S240). Displayed on the screen and operated by hand movement like a computer mouse operation.

  FIG. 13 shows an example of transmission data of a transmission signal according to the second embodiment of the present invention. The transmission code of the remote control is generally composed of a custom code indicating the model or manufacturer and a data code indicating the operation. For example, as shown in FIG. Specified code ”+“ 3-byte data ”, for example, the XY coordinate position moved when moved while holding the hand according to the data format of the serial mouse. The left button was pressed when one finger was raised. The left button was released when the finger was returned from the standing position, the right button was pressed when the two fingers were raised, the right button was released when the finger was returned from the standing position, Thus, when there is a change, a signal to be displayed on the control target device is transmitted as a transmission signal 115 to a television or a recorder. By doing so, the same operation as the mouse operation on the browser of the personal computer can be realized on the browser of the television or recorder.

  In recent years, the Internet environment has become widespread, and even TVs and recorders can be connected to the Internet. In general, browsers on TVs and recorders simply move the focus position with the cross key button on the remote control, and you can Couldn't be said to easily move the pointer and click. However, in the remote operation device and the remote operation method configured as described above, the button on / off information and the movement amount, which are basic functions of mouse operation, are detected by the manual operation detection means and transmitted to the display device 112 and the like. Since the movement speed and movement amount of the hand are reflected and displayed as they are in the pointer movement, the operation can be easily performed with a gesture like a mouse of a personal computer, and the operability is improved.

  Here, the browser is mounted on the display device 112 and the recording / playback device 113 and transmits the gesture operation. However, even when the remote operation device 111 itself is mounted with the browser, naturally the gesture operation can be easily performed. Become.

  In addition, examples of the use of the remote control device and remote control method of the present invention include improvement in the operability of devices that are multifunctional and have a large number of buttons on the remote controller, such as a digital TV receiver and HDD built-in DVD recorder. It has the feature that it can adapt to.

It is an apparatus block diagram of the remote control system which shows Embodiment 1 of this invention. It is a functional block diagram of the remote control device which shows Embodiment 1, 2 of this invention. It is a hardware block diagram of the remote control device which shows Embodiment 1, 2 of this invention. It is a figure of the display mode transition which shows Embodiment 1 of this invention. It is a figure of the display mode transition and apparatus state transition which show Embodiment 1 of this invention. It is a figure of the display mode transition and apparatus state transition which show Embodiment 1 of this invention. It is a figure of the example of button operation by the gesture which shows Embodiment 1 of this invention. It is a flowchart of the gesture detection process which shows Embodiment 1, 2 of this invention. It is a figure explaining the area | region determination of the gesture detection process which shows Embodiment 1, 2 of this invention. It is a figure explaining the outline direction calculation of the gesture detection process which shows Embodiment 1, 2 of this invention. It is a figure explaining the center outline distance calculation of the gesture detection process which shows Embodiment 1, 2 of this invention. It is an apparatus block diagram of the remote control system which shows Embodiment 2 of this invention. It is a figure of the example of transmission data which shows Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 111 Remote operation apparatus 112 Display apparatus 113 Recording / reproducing apparatus 114 Connection cable 115 Transmission signal 116 Imaging means 117 Liquid crystal touch panel part 118 Hardware button 121 Operation display means 122 Imaging means 123 Hand area detection means 124 Hand operation detection means 125 Communication means 126 Control Means 131 CPU
132 Memory 133 Liquid Crystal Panel 134 Touch Panel 135 Liquid Crystal Touch Panel 136 Signal Transmission / Reception Unit 137 Sound Generator 138 Imaging Means 141 Television Mode Button 142 Reservation Mode Button 143 Playback Mode Button

Claims (14)

Operation display means for displaying an operation button or menu for remote operation of the control target device;
Imaging means for imaging an operator operating based on the display of the operation display means;
Hand region detecting means for detecting a hand region from an image photographed by the imaging means;
Hand operation detection means for detecting an operation by the operator based on the shape and movement of the hand area detected by the hand area detection means and the display of the operation display means;
A remote operation device comprising: a control unit that transmits an operation signal detected by the manual operation detection unit to the control target device using a communication unit.   The remote control device according to claim 1, wherein the control unit displays the detected operation on the operation display unit.   The remote control device according to claim 1, wherein the control unit further transmits a signal for displaying the detected operation on the control target device. The hand region detecting means includes
Binarization correction means for extracting a region based on a predetermined signal from the image and performing binarization by removing noise;
Distance calculating means for calculating a point within the binarized image area obtained by the binarized correcting means and having the maximum distance to the contour of the binarized image area;
A center-weighted correction unit that calculates a correction value that is larger as it is closer to the center of the image and closer to the edge of the image with respect to the maximum distance point from the distance calculation unit
The remote control device according to claim 1, further comprising: a hand region determination unit that determines the hand region and the center point based on a calculation result of the center weight correction unit. The manual operation detection means includes
Contour direction calculating means for determining the direction of the contour along the contour line of the hand region;
Hand operation determining means for determining the shape of the hand region from the direction of the contour determined by the contour direction calculating means, and detecting the operation of the hand based on the shape and movement and the display of the operation display means. The remote control device according to claim 1. The manual operation detection means includes
A center contour distance calculating means for calculating a distance between the contour of the hand region and a center point of the hand region;
The shape of the hand region is determined from the relationship between the maximum point and the minimum point of the distance calculated by the center contour distance calculating means, and the operation of the hand is detected based on the shape, the movement, and the display of the operation display means. The remote operation device according to claim 1, further comprising a manual operation determination unit. The manual operation detection means includes
7. The remote operation according to claim 5, wherein a selection operation is detected from the shape of the hand region, and a movement operation is detected from the shape of the hand region and a movement of a center point of the hand region. apparatus. In a remote control device for remote control of a control target device,
An operation display step of displaying the operation button or menu for remote operation on an operation display means;
An imaging step of imaging with an imaging means an operator operating based on the display of the operation display means;
A hand region detection step of detecting a hand region from the image captured in this imaging step;
A hand operation detection step of detecting an operation by the operator based on the shape and movement of the hand region detected in the hand region detection step and the display of the operation display means;
A remote operation method comprising a communication step of transmitting an operation signal detected in the manual operation detection step to the control target device using a communication means.   The remote operation method according to claim 8, further comprising the operation display step of displaying the detected operation on the operation display means.   The remote operation method according to claim 8, wherein the communication step further transmits a signal for displaying the detected operation on the device to be controlled. The hand region detection step includes
A binarization correction step of extracting from the image based on a predetermined signal based on a predetermined signal, performing noise removal, and binarizing;
A distance calculating step for calculating a point within the binarized image area obtained in the binarized correction step and having a maximum distance to the contour of the binarized image area;
A center emphasis correction step for calculating a correction value that is larger as it is closer to the center of the image and closer to the edge of the image, relative to the maximum distance point from the distance calculation step
The remote operation method according to claim 8, further comprising a hand region determination step of determining the hand region and the center point based on a calculation result of the center weight correction step. The manual operation detection step includes:
A contour direction calculating step for obtaining a contour direction along the contour line of the hand region;
A hand operation determining step of determining the shape of the hand region from the contour direction obtained in the contour direction calculating step, and detecting a hand operation based on the shape, movement, and display of the operation display means. The remote control method according to claim 8. The manual operation detection step includes:
A center contour distance calculating step for calculating a distance between the contour of the hand region and a center point of the hand region;
The shape of the hand region is determined from the relationship between the maximum point and the minimum point of the distance calculated in the center contour distance calculating step, and the operation of the hand is detected based on the shape, the movement, and the display of the operation display means. The remote operation method according to claim 8, further comprising a manual operation determination step. The manual operation detection step includes:
14. The remote operation according to claim 12, wherein a selection operation is detected from the shape of the hand region, and a movement operation is detected from the shape of the hand region and the movement of the center point of the hand region. Method.

Images