JP2010283549A - Gesture-based remote control system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a gesture-based remote control system controlling at least one piece of controlled electronic equipment. <P>SOLUTION: A movement control command is obtained by recognizing a user's image in a camera module 300 by an image recognition module 100. A key control command is obtained from the movement control command and key code information by the image recognition module 100. The key control command is transmitted to a radio transmitter 400 by the image recognition module 100. The key control command is transmitted to a controlled electronic application by the radio transmitter in order to control the controlled electronic application to make gesture-based remote control. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

      The present invention relates to remote control systems, and in particular to gesture-based remote control systems.

      It is inconvenient and cumbersome to operate many different home appliances using different corresponding remote controls. Thus, many different key code information databases are incorporated into the same remote control that operates various home appliances, so that a single remote control is used with a mode switch key if it is sufficient for mode selection. In addition, some premium remote controls are designed to emulate different brands of remote control key codes to overcome the shortage of key code information databases. However, it is inconvenient to operate the home appliance if the user forgets where the responding physical remote control is placed.

      To remedy the aforementioned drawbacks, the present invention provides a gesture-based remote control system that controls at least one controlled electronics.

      To achieve the aforementioned objectives, a gesture-based remote control system includes a camera module, an image recognition module, a wireless transmitter, and main control electronics. The image recognition module is electrically connected to the camera module. The wireless transmitter / receiver is electrically connected to the image recognition module. The main control electronic device is detachably connected to the image recognition module and has a monitor. The movement control command is obtained by recognizing the user image in the camera module by the image recognition module. A key control command is obtained from the movement control command and the key code information by the image recognition module. The key control command is transmitted to the controlled electronic application by the wireless transmitter to control the controlled electronic application.

      The features of the invention believed to be novel are set forth with particularity in the appended claims. The present invention itself, however, can best be understood by reference to the following detailed description of the invention, which describes exemplary embodiments of the invention taken in conjunction with the accompanying drawings, in which:

1 is a block diagram of a gesture-based remote control system according to the present invention. FIG. It is a schematic diagram which uses an image difference method. It is a schematic diagram of a some division | segmentation section. It is a schematic diagram which searches the direction of a movement area | region. It is a typical front view which shows the position gesture which lifts both arms outside and shakes both arms up and down. It is a typical front view which shows a user's position and the corresponding position of a simulated key press key. It is a typical bottom view which shows the cancellation gesture which lifts both arms ahead and shakes both arms to the left and right. It is a flowchart which performs a menu procedure. It is a flowchart which performs a position procedure. It is a flowchart which performs a cancellation procedure. 7 is a flowchart for executing an increment procedure. It is a flowchart which performs a decrement procedure. It is a flowchart which performs a click procedure. and, 3 is a flow diagram for implementing a gesture-based remote control system.

      As described above, when the user forgets where the responding physical remote control is located, it is inconvenient to operate the home appliance. In order to overcome the aforementioned drawbacks, the present invention provides a gesture-based remote control system without physical remote control to operate electrical equipment. A gesture-based remote control system provides a camera module that captures different images of a user's gesture. A gesture image (image of the user's gesture) is processed to generate a control signal corresponding to the button of the physical remote control, and this control signal is provided to control the electrical equipment. In a more detailed description, an electronic device (such as a television) with an on-screen display (OSD) is provided to be a universal remote control. Many different key codes for various remote controls are simulated, or a key code database is used directly by a general purpose remote control. In addition, the contents of the action menu are displayed directly on the monitor, and the action menu is operated to control various electronic devices without using any physical remote control.

      Reference is made to FIG. 1, which is a block diagram of a gesture-based remote control system according to the present invention. The gesture-based remote control system includes a camera module 300, an image recognition module 100, a wireless transmitter 400, an infrared port 500, and main control electronics 200. The image recognition module 100 is electrically connected to the camera module 300, the wireless transmitter / receiver 400, the infrared port 500, and the main control electronic device 200, respectively.

      The image recognition module 100 has key code information of at least one controlled electronic device (not shown), and this key code information is stored in the image recognition module 100. The image recognition module 100 further includes a digital image processor 102, a microprocessor 104, an A / D converter 114 and an erasable PROM (EEPROM) 112. The microprocessor 104 is electrically connected to the wireless transmitter 400, the infrared port 500, the main control electronics 200, the digital image processor 102, the A / D converter 114, and the erasable PROM (EEPROM) 112. The digital image processor 102 is electrically connected to the camera module 300.

      More specifically, the image recognition module 100 is provided to recognize a user image in the camera module 300 in order to obtain a movement control command. Further, a key control command is obtained by the image recognition module 100 from the movement control command and the key code information. The image recognition module 100 transmits a key control command to the wireless transmitter 400. The key control command is also transmitted to the controlled electronic application by the wireless transmitter (400) to control the controlled electronic application.

      The image recognition module 100 receives key code information transmitted from the controlled electronic device through the infrared port 500. The key code information is translated and then stored in an erasable PROM (EEPROM) 112. In addition to the erasable PROM (EEPROM) 112, hundreds of well-known key code databases of the controlled electronics 200 can be stored in the erasable PROM (EEPROM) 112. The wireless transmitter 400 can transmit different types of IR or RF key control commands to control the controlled electronics according to the type of controlled electronics. If the main control electronics 200 is a controlled electronics (such as a television), the image recognition module 100 sends key control commands directly to the main control electronics 200 through different transmission interfaces (such as I2C, SPI or UART). Can be sent. In addition to digital signals, the image recognition module 100 can also send analog signals that are processed by the A / D converter 114 to the main control electronics 200.

      The controlled electronics can be a television, a DVD player, an air conditioner, or a computer. Main control electronics 200 (such as a television) has a monitor 202 and an on-screen display (OSD) 204. The key control command includes a start command, a cancel command, an increment command, a decrement command, and a click command. For example, the image recognition module 100 is detachably connected to the main control electronic device 200 via a serial port. The image recognition module 100 sends video signals to the main control electronics 200 to display them on the monitor 202 for showing operational messages during the operational process.

      The gesture-based remote control system determines the user's gestures for position gestures, click gestures, slide gestures, and cancel gestures. A detailed description of the different gestures is given as follows:

      1. Position gesture. A remote control system based on a general purpose simulated gesture has an electronic device (such as a television) with an on-screen display (OSD) function. The operation menu is initially set to an invalid state after the gesture-based remote control system is activated. First, the optimal operating position for the user is automatically detected, and the main menu is activated when the user lifts both arms outward and swings both arms up and down (shown in FIG. 5). Displayed on the monitor of the electronic device. The operation position of the simulated key press key is located at four different positions. The first position is located above the user's head, the second position is located outside both of the first positions, the third position is located outside both of the second positions, and The fourth position is near the left and right hands of the user (shown in FIG. 6). More specifically, the function selection block is located at the first position, the second position, and the third position. In addition, the function adjustment block is positioned at the fourth position, and the function adjustment area is operated to switch (rotate) the operation menu or transmit the function key.

      2. Click gesture. The click gesture is applied to the function selection area. The selection operation is performed such that one of the simulated key pressing keys located in the function selection area is clicked once. In addition, when one of the simulated key press keys is clicked again, the selection operation is canceled. Therefore, the simulated key press key is called a toggle key. Further, when another simulated key press key is clicked, the current simulated key press key is automatically released. Thus, only one of the simulated key press keys is activated at the same time, which is similar to a radio button in a computer GUI menu.

      3. Slide gesture. The slide gesture is applied to the function adjustment area. The right simulated key supports only gestures that swing the right arm, and the left simulated key supports only gestures that swing the left arm. If none of the function selection blocks are selected, the action menu is switched (rotated) to the right or left. When one of the function selection blocks is selected and the user swings his (female) arm to the right, a forward function key is transmitted. Conversely, when one of the function selection blocks is selected and the user swings his (woman) arm to the left, a reverse function key is transmitted. For example, volume adjustment is taken as an example. When the volume control function (the operation function is set in one of the function selection blocks) is selected, and the user swings his (woman) arm to the right, the volume is increased. Conversely, when the volume control function is selected and the user swings his (woman) arm to the left, the volume is lowered. In addition, if the user swings his (female) arm to the left and right, forward direction function keys and reverse direction keys are also transmitted, respectively. It is not limited to the operation described above.

      4). Cancel gesture. If the user lifts both arms forward and swings both arms to the left and right (shown in FIG. 7), the cancel gesture can be operated to return to the previous menu or close the current menu. .

      The above-mentioned operation menu of the remote control system based on the general-purpose simulation gesture can be switched to a main menu and a sub menu. A main menu is provided to switch between controlled electronics device options and remote control system configuration options. In the main menu, when the user swings his (female) arm to the left or right, the function adjustment block is activated to switch (rotate) the instrument option. More particularly, the equipment options for controlled electronic equipment are a television, a DVD player, an air conditioner, a computer or others. When one of the device options of a controlled electronic device is selected, the corresponding submenu is activated, i.e. opened. Further, when the cancel gesture is operated, the main menu can be closed. A submenu is provided to switch the operation option of the corresponding device option. In the submenu, the operation option in the function selection block can be selected to operate, and the operation of the operation option can be canceled. If none of the function selection blocks are selected and one of the simulated keystrokes in the function adjustment block is clicked, the operation options (such as volume control, channel selection or color adjustment) of the controlled electronics are It can be switched (rotated). In addition, when one of the function selection blocks is selected and one of the simulated key press keys in the function adjustment block is clicked, a forward function key or a reverse function key is transmitted. When the cancel gesture is actuated (shown in FIG. 8), the submenu can be actuated to return to the main menu.

      When another user wants to operate the gesture-based remote control system, the main menu must be closed, and the position gesture (the user lifts both arms to the left and right and raises both arms Shake up and down) is actuated again. Thus, only one user can operate the remote control system at the same time, i.e. only one user can be captured in the visible region by the camera module.

      Referring to FIG. 14, this is a flowchart for implementing a gesture-based remote control system. A gesture-based remote control system is provided for controlling a controlled electronic device by detecting a user's gesture. The controlled electronics can be a television, DVD player, air conditioner, computer or others. First, a user image is captured by the camera module (S10). Thereafter, the image is adjusted to obtain an adjusted image (S20). Thereafter, an adjusted moving image of the adjusted image is calculated using an image difference method (S30). Thereafter, a moving image in the adjusted image is detected (S40). Thereafter, a moving area in the moving image is defined (S50). Thereafter, a corresponding control signal is generated according to the movement region (S60). Finally, a control signal is transmitted by the wireless transmitter to control the controlled electronic device (S70). A detailed description of operating the gesture-based remote control system is given as follows.

      Adjusting the user gesture image S20 to obtain an adjusted image includes the following sub-steps: (1) Adjusting the processed size of the user gesture image. (2) Convert the color of the user's gesture image (from a 24-bit full color image to an 8-bit gray level image). And (3) filter speckle noise in the user's gesture image. More specifically, speckle noise in the user's gesture image can be filtered by an image low pass filter.

      Step S30, an adjusted moving image of the adjusted image is calculated using an image difference method. Referring to FIG. 2, this is a schematic diagram using the image difference method. To obtain better performance, three consecutive gesture images are supplied to calculate the adjusted moving image. Three successive gesture images (images of the user's gesture) are respectively the current gray level image 12, the previous gray level image 11 before the current gray level image 12, and the previous gray level image 11 before It is a gray level image 10 that precedes in advance. A first gray level threshold and a second gray level threshold are set to convert the gray level image to a binary image. First, the current gray level image 12 is subtracted by the preceding gray level image 11 to obtain a first gray level image (not shown). Thereafter, the gray value of each pixel of the first gray level image is compared to a first gray level threshold. When the pixel gray value is greater than or equal to the first gray level threshold, the pixel is set as a light pixel, and conversely, when the pixel gray value is less than the first gray level threshold, the pixel is set as a dark pixel. The Therefore, the first binary image 13 is composed of bright pixels and dark pixels. Similarly, the preceding gray level image 11 is subtracted by the preceding gray level image 10 to obtain a second gray level image (not shown). Thereafter, the gray value of each pixel of the first gray level image is compared to a second gray level threshold. When the pixel gray value is greater than or equal to the second gray level threshold, the pixel is set as a light pixel, and conversely, when the pixel gray value is less than the second gray level threshold, the pixel is set as a dark pixel. The Therefore, the second binary image 14 is composed of bright pixels and dark pixels. Finally, a logical AND operation is performed between the first binary image 13 and the second binary image 14 to generate a third binary image 15 that is adjusted and moved. It is an image. Therefore, the position of the image of the user's gesture can be detected.

      A detailed description of step S40 is given as follows. Initially, the third binary image 15 is divided into a plurality of divided sections (shown in FIG. 3). A movement threshold is set to determine whether each of the divided sections is a movement section. In a preferred embodiment, the divided section is set as a bright section when the sum of the moving pixels of the divided section is greater than the moving threshold, ie, the divided section is considered a moving section. On the other hand, when the total number of moving pixels in the divided section is less than the moving threshold, the divided section is set as a dark section, i.e., the divided section is not considered a moving section. For example, an image of 160 * 120 pixels is divided into 192 (16 * 12 = 192) divided sections, ie, each of the divided sections has 100 ((160/16) * (120/12) = 100) pixels. . It is assumed that the movement threshold is set to 50. A divided section is a moving section if the sum of the moving pixels in one divided section is greater than the movement threshold. Thus, large or slight movements can be filtered to reduce the possibility of manipulation errors.

      A detailed description of step S50 is given as follows. As shown in FIG. 4, the coordinate boundary of the moving region in the moving image is determined from (LTX, LTY) to (RBX, RBY). When a moving section is first detected from top to bottom and left to right in the moving image, the top edge of the moving area, LTY, is set. In addition, the bottom end (RBY), left end (LTX), and right end (RBX) of the moving area are set in a similar manner. When the coordinate boundary of the moving area is from (0,0) to (0,0), the moving section is not detected or the area of the moving section is too small. Therefore, when the elapsed time exceeds the set time, the operation menu (including the main menu and sub menu) is automatically closed.

      As described above, the gesture-based remote control system determines a user's gesture with respect to a position gesture, a click gesture, a slide gesture, and a cancel gesture. A click gesture is a time-independent gesture, however, a slide gesture, a position gesture, or a cancel gesture is a time-dependent gesture. In order to recognize the type of these gestures, the latest coordinates and size of the moving area need to be recorded. When the moving area overlaps the click regulation block, it is recognized that the click gesture supplies a click command. As the moving area moves sideways (to the left or to the right), a slide gesture is recognized. More specifically, the moving area continuously moves sideways, i.e. the movement continuously moves to the left or to the right. If the moving area continuously moves to the right in the function adjustment block, an increment command is given. On the other hand, if the moving area moves to the left in the function adjustment block, a decrement command is given. A cancel gesture is generated when the moving region continuously changes horizontally (shown in FIG. 7). When the moving region changes from horizontal to vertical or from vertical to horizontal (shown in FIG. 5), a position gesture is generated. However, if the moving area is too large or the moving area continuously changes in the vertical direction, the moving object generates an abnormal disturbance through the lens of the camera module. Also, other undefined or unrecognized gesture actions are invalid.

      A detailed description of step S60 is given as follows:

      1. Position gestures can be operated in both the main menu and submenu. In addition, a position gesture is detected according to the movement region to generate an actual motion region. The main menu is opened and displayed on the monitor. The position procedure is shown in FIG.

      2. A cancel gesture can be operated in both the main menu and the submenu to return to the previous menu or close the current menu. The cancellation procedure is shown in FIG.

      3. Incremental slide gestures (further referred to as incremental gestures) can be operated in both the main menu and submenu. When the increment gesture is actuated in the main menu, the device option in the function selection block is switched to the right (rotated). If the increment gesture is activated in the submenu and one of the motion options is not selected, the motion option is switched (rotated) to the right. In addition, when the increment gesture is activated in the submenu and one of the function options is selected, the increment function is activated. The increment slide procedure is shown in FIG.

      4). A decrement slide gesture (further referred to as a decrement gesture) is operated in the same way as an increment gesture. The difference between an increment gesture and a decrement gesture is that the switched (rotated) direction and some amount is opposite. The decrement slide procedure is shown in FIG.

      5. Click gestures can be operated in both main menus and sub-menus. If the click gesture is operated in the main menu and one of the function selection blocks is selected correctly, the selection of the function selection block is valid. If the click gesture is actuated in the submenu, the selected action option is enabled, and furthermore, if the click gesture is actuated again, the selected action option is invalidated. Furthermore, if another operation option is selected, the current operation option is closed. Therefore, only one of the operation options is operated at the same time. The click procedure is shown in FIG.

      In conclusion, the present invention has the following features:

      1. The user's skin color, clothing and decoration, and the complexity of the environmental background are not limited to operating a gesture-based remote control system. The user can also manually operate a non-contact mouse device without holding any object of special color or pattern, portable lighting equipment, wearing any special data glove, or operating with a special gesture. It can be operated.

      2. A gesture-based remote control system is provided to combine a traditional TV or digital TV with a conventional TV or digital TV to make it a multifunctional general purpose remote control. The contents of the action menu can be displayed directly on the TV monitor, and the action menu can control various electronic devices by simply using the user's gestures without any physical remote control. Be operated.

      3. Since the action position of the action menu is located close to the user, a defined action option can be easily selected. In addition, a simple arm motion such as swinging the arm to the left or right can realize an operation of switching (rotating) the operation option and transmitting the forward function key and the reverse function key.

      4). A circular menu is adopted that includes more action options, is even easier to operate and is intuitive.

      5. Extremely large moving areas are automatically filtered to exclude false moving objects.

DESCRIPTION OF SYMBOLS 10 Previous gray level image 11 Previous gray level image 12 Current gray level image 13 First binary image 14 Second binary image 15 Third binary image 100 Image recognition module 102 Digital image processor 104 Microprocessor 112 EEPROM
114 A / D Converter 200 Main Control Electronics 202 Monitor 204 OSD
300 Camera module 400 Wireless transmitter 500 Infrared port

Claims (11)

A remote control system based on gestures,
A camera module (300);
An image recognition module (100) electrically connected to the camera module (300);
A wireless transmitter (400) electrically connected to the image recognition module (100);
A main control electronic device (200) having a monitor (202) and detachably connected to the image recognition module (100),
A movement control command is obtained by recognizing a user image in the camera module (300) by the image recognition module (100), and a key control command is obtained by the image recognition module (100). Obtained from key code information, the key control command is transmitted to the wireless transmitter (400) by the image recognition module (100), and the key control command is transmitted wirelessly to control a controlled electronic application. Transmitted by the machine (400) to the controlled electronic application.
The gesture-based remote control system of claim 1, wherein the main control electronics (200) is a television.
3. The gesture-based remote control system of claim 2, wherein the television has an on-screen display (OSD) (204).
2. The gesture-based remote control system according to claim 1, wherein the controlled electronic device is a television, a DVD player, an air conditioner or a computer.
The gesture-based remote control system of claim 1, wherein the image recognition module (100) further comprises an erasable PROM (EEPROM) (112).
The gesture-based remote control system of claim 1, further comprising an infrared port (500) electrically connected to the image recognition module (100), wherein the image recognition module (100) is the A system for receiving the key code information transmitted from the controlled electronic application via an infrared port (500).
The remote control system based on the gesture of claim 1, wherein the key control command comprises a start command, a cancel command, an increment command, a decrement command, and a click command.
The remote control system based on the gesture of claim 1, wherein the recognition module (100) is detachably connected to the main control electronics (200).
9. The gesture-based remote control system according to claim 8, wherein the image recognition module (100) is connected to the main control electronics (200) via a serial port.
The gesture-based remote control system of claim 1, wherein the image recognition module (100) further comprises:
A digital image processor (102) electrically connected to the camera module (300);
A microprocessor (104) electrically connected to the digital image processor (102);
An A / D converter (114) electrically connected to the microprocessor (104).
      The gesture-based remote control system according to claim 1, wherein the key code information is stored in the image recognition module (100).