TWI518551B - An electronic device controlling system and method thereof - Google Patents

Description

Electronic device control system and method thereof

The present invention relates to an electronic device control system and method thereof, and more particularly to a system and method for controlling an electronic device using a user's limb motion.

Nowadays, with the progress of life, technology is no longer an instrument only for professionals, but a tool that is closely related to people's lives. Therefore, good human-computer interaction has been the subject of continuous improvement. From the use of general buttons, to touch, and voice control, it has caused an explosive revolution in information, enabling more people to use technology to improve their lives. Nowadays, image processing and human-computer interaction are more popular in recent years. The products included in the technology include Nintendo's Wii, which uses infrared light as a reading lens and combines three-axis acceleration to reach the user. The judgment and identification of limb movements, followed by Xbox360's Kinect and PlayStation3's MOVE and other products. The automobile industry, Smart TV and mobile phones have gradually moved toward this concept and launched related products one after another, and are developing towards civilians and humanization. Looking at the combination of image processing and machine interaction does have its potential for development and the value of research.

However, the human-machine interface products of the prior art require a dedicated host and are equipped with peripheral equipment. The price of the host and the peripheral equipment is still expensive, and the use is only for entertainment. Generality is not high.

In response to the foregoing problems, the present invention provides an electronic device control system for constructing a human-machine interface in an embedded system of an ARM (Advanced RISC Machine) platform, by identifying a user's first motion image and placing the user The limb motion image is used as an input command to control and operate the electronic device. The method comprises: an image pre-processing module, an image recognition module and an image comparison module.

The image pre-processing module is configured to pre-process the first motion image to generate a second motion image; the image recognition module is electrically connected to the image pre-processing module to define an action part of the second motion image, and track And identifying an action corresponding to the action part, and then outputting an action instruction formed by the action; and the image comparison module is electrically connected to the image recognition module, and the image comparison module includes an action instruction database and an action instruction data The library stores a preset action command, and the image comparison module is configured to compare whether the action command conforms to the preset action command. If the comparison result is consistent, outputting one of the preset action command control commands to control the electronic device.

The action part is any actionable part of the user's limb, such as the hand, the foot, the eye or the lip; the action includes a palm movement track and a palm gesture, and the action command includes a palm movement track instruction and a Hand gestures.

The present invention also provides an electronic device control method, comprising the steps of: S1: receiving a first motion image of a user; S2: pre-processing the first motion image to generate a second motion image; and S3: An action part is defined in the image, and one action corresponding to the action part is recognized, and then an action instruction formed by the action is output; and S4: the action instruction is compared with a preset action instruction; wherein, if the result is compared If it matches, the output controls the instruction corresponding to one of the preset action instructions.

The action part is any actionable part of the user's limb, such as the hand, the foot, the eye or the lip; the action includes a palm movement track and a palm gesture, and the action command includes a palm movement track instruction and a Hand gestures.

Compared with the prior art, the electronic device control system and method provided by the present invention are built into an embedded system of an ARM (Advanced RISC Machine) platform, so that the electronic product with the receiving end is attached. The invention can be used to replace the function of the general remote controller or the touch control by a user only by using the gesture action at a long distance, and achieve the convenience, time saving and low cost.

1‧‧‧Electronic device control system

11‧‧‧Image capture device

12‧‧‧Image pre-processing module

13‧‧‧Command output module

14‧‧‧Image recognition module

16‧‧‧Image comparison module

2‧‧‧Electronic device control method

C1‧‧‧Control Instructions

S1~S5‧‧‧Steps

Substeps S21~S24, S31~S33‧‧

Figure 1 depicts a functional block diagram in accordance with an embodiment of the present invention.

Figure 2 depicts a flow chart of steps S1 - S4 in accordance with an embodiment of the present invention.

Figure 3 depicts a flow chart of sub-steps S21-S24 in accordance with an embodiment of the present invention.

Figure 4 is a schematic diagram showing a time difference subtraction method in accordance with an embodiment of the present invention.

Figures 5A through 5B illustrate schematic diagrams of erosion operations in accordance with an embodiment of the present invention.

6A to 6B are diagrams showing an expansion operation according to an embodiment of the present invention.

7A through 7B are schematic diagrams showing the communication structure according to an embodiment of the present invention.

Figure 8 depicts a flow chart of sub-steps S31-S33 in accordance with an embodiment of the present invention.

In order to make the invention more apparent, the following detailed description of the invention and the examples thereof are included to provide a better understanding of the invention.

This description is only for the purpose of illustrating the essential elements of the invention, and is only intended to illustrate the possible embodiments of the invention, but the description of the specification should not limit the scope of the technical nature of the claimed invention. The present invention is not limited to the specific methods, procedures, functions, or means unless the scope of the invention is specifically excluded. It is also to be understood that the presently described embodiments are merely possible embodiments of the present invention, and any methods, procedures, functions or means similar or equivalent to those described herein may be employed in the practice or testing of the invention.

Unless otherwise defined, all technical and scientific terms used in the specification have the same meaning meaning Although any methods and means similar or equivalent to those described in the present specification can be used in the practice or testing of the present invention, the presently described embodiments are merely exemplary methods, procedures, and related materials.

Furthermore, one or more of the numbers mentioned in the specification include the number itself. It should be understood that the present disclosure discloses certain methods and processes for performing the disclosed functions. There are a variety of structures related to the disclosed structures that perform the same functions, and the above structures generally achieve the same result.

First, referring to FIG. 1, FIG. 1 depicts a functional block diagram in accordance with an embodiment of the present invention. The present invention provides an electronic device control system 1 for constructing a human-machine interface in an embedded system of an ARM (Advanced RISC Machine) platform, and recognizing a user's limb motion by recognizing a first motion image of a user Image as an input command to control and operate Electronic device. The image capture device 11 includes an image capture device 12 , an image pre-processing module 12 , an instruction output module 13 , an image recognition module 14 , and an image comparison module 16 .

The image capturing device 11 is electrically connected to the image pre-processing module 12 for use in capturing The first motion image is output to the image pre-processing module 12; the image pre-processing module 12 is configured to pre-process the first motion image to generate a second motion image; the image recognition module 14 The image processing pre-processing module 12 is configured to define an action part of the second motion image, and track and recognize one action corresponding to the action part, and then output an action instruction formed by the action; the image comparison module 16 The image matching module 14 is electrically connected to the image matching module 16, and the image comparison module 16 includes an action instruction database. The action command database stores a preset action command, and the image comparison module 16 is configured to compare whether the action command conforms to the preset action. The instruction outputs a control command C1 corresponding to one of the preset action commands to control the electronic device; the command output module 13 is electrically connected to the image comparison module 16 for transmitting the control command C1 to the electronic device. Control of the electronic device.

Wherein, the image capturing device 11 is a camera; the action portion is the use Any actionable part of the limb, such as the hands, feet, eyes or lips. The action corresponding to the action part includes a palm movement track and a palm gesture, and the action command formed by the action includes a palm movement track command and a palm gesture command.

Referring next to FIG. 2, FIG. 2 depicts steps in accordance with an embodiment of the present invention. S1~S4 flow chart. The present invention also provides an electronic device control method 2, which includes the following steps: S1: receiving a first motion image of a user; S2: pre-processing the first motion image to generate a second motion image; S3: The action image defines an action part, and identifies one action corresponding to the action part, and then outputs an action instruction formed by the action; and S4: the action finger Aligning with a preset action instruction; wherein, if the comparison result is met, the output corresponds to one of the preset action instructions to control the instruction.

Step S1 uses the image capturing device 11 to capture the first action of the user. The image is transmitted to the image pre-processing module 12 to perform step S2: pre-processing the first motion image to generate a second motion image.

In the following, the hand as the action part is taken as an example to illustrate how to act on the first action. Like the pre-processing to generate a second motion image. Referring to FIG. 3, FIG. 3 depicts a flowchart of sub-steps S21-S24 according to an embodiment of the present invention. Step S2 includes sub-steps S21-S24, which are respectively S21: removing the static background in the first motion image and leaving at least one motion object image; S22: removing noise in at least one motion object image; S23: strengthening At least one motion object image; and S24: numbering at least one motion object image to form and output a second motion image including at least one motion object image.

Referring to FIG. 4, FIG. 4A depicts a time difference according to an embodiment of the present invention. Schematic diagram of the subtraction method. The first motion image captured by the image capturing device 11 includes a static background image and a motion image of the user's motion object. In sub-step S21, the Temporal Difference is used to remove the static in the first motion image. Background to leave at least one moving object image. The Temporal Difference method splits the first motion image into a plurality of frames, then subtracts the current frame from the previous frame, and binarizes the obtained values. At least one moving object image is captured in the first motion image to achieve the effect of removing the static background (as shown in FIG. 4). Sub-step S22 is then performed: removing noise in the image of at least one moving object.

Please refer to FIG. 5A to FIG. 5B, and FIG. 5A to FIG. 5B are drawn according to the present invention. A schematic diagram of the erosion operation of an embodiment. Since the static background in the first motion image is not completely static, and under the light and shadow changes, the time difference subtraction method will often appear as noise or block noise as moving, as shown in Figure 5A. . In the sub-step S22, an erosion operation (Erosion) is used to remove noise in at least one moving object image, as shown in FIG. 5B.

Please refer to FIG. 6A to FIG. 6B, and FIG. 6A to FIG. 6B are drawn according to the present invention. A schematic diagram of the expansion operation of an embodiment. Next, sub-step S23 is performed to remove at least one moving object image of the noise to enhance at least one moving object image. Because at least one moving object image undergoes an erosion operation, in addition to removing noise, the block of interest in at least one moving object image appears to shrink, as shown in FIG. 6A. Therefore, it is necessary to restore these blocks to the size before the erosion operation, as shown in Figure 6B.

Next, the last sub-step S24 of the image pre-processing will be performed: the at least one motion The object images are numbered to form and output the second motion image including the image of the at least one motion object. Referring to FIG. 7A to FIG. 7B, FIG. 7A to FIG. 7B are diagrams showing the connection structure indication according to an embodiment of the present invention.

After a series of previous processes, at least one action as shown in FIG. 7A can be obtained. Object image, in which three groups of blocks represented by white pixels are images of moving objects, we need to mark each connected white pixel block separately to output image pre-processing, and include at least one moving object image. The second motion image. The invention will use the Connected Component Labeling to complete the block labeling. The connectivity structure designates the attributes of different blocks into different categories, and the blocks that are connected to each other are given the same number, as shown in the numbers 1 to 3 of Figure 7B, and the images of the numbered action objects are limb images. Candidate.

Referring next to FIG. 8, FIG. 8 depicts substeps in accordance with an embodiment of the present invention. Steps S31~S33. After the second motion image is acquired, step S3 is performed: an action portion is defined in the second motion image, and one motion corresponding to the motion portion is recognized, and an action command formed by the motion is output. The step S3 includes the following sub-steps: S31: removing the number of the non-limb image in the image number of the at least one action object, and further defining the action part from the image of the at least one action object; S32: tracking the action part And S33: identifying an action formed by the action portion, and outputting an action command formed by the action.

In the step S31, the present invention successively utilizes the similarity ratio method to remove the blocks of the at least one action object image number in the second motion image that are not the user's limb image, that is, the numbers 2 and 3 in FIG. 7B are eliminated. Block, and use Harris detection to further define the user's action location, which is number 1 in Figure 7B. To this end, the definition of the action site has been completed. Where the action part is the hand or the palm

Sub-step S32 is then performed: the action portion is tracked. The invention utilizes the Histogram of Oriented Gradients (HOG) and the Mean Shift method to track the action parts. For example, the palm portion of the action portion in the first motion image is undergoing a clockwise bypass motion or any other motion. In the present step, the present invention tracks the position of the action portion in each frame. Among them, the gradient features in HOG are mainly used, and the grayscale feature in the average displacement method is supplemented.

Finally, sub-step S33 is performed: identifying an action formed by the action part, and outputting an action command formed by the action. The present invention recognizes the action performed by the action site by the Temporal Difference method and the scan projection method. For example, the time difference subtraction method is used to identify the movement trajectory formed by the movement of the action part (the palm part), or whether the palm part is made to make a gesture of making a fist; if the action part (palm part) is made with a finger Compared with the digital gesture action, the present invention uses the scanning projection method to identify the number compared to the action portion (the palm portion). Finally, an operation command including an action part movement track command and an action part gesture command is output.

After the above sub-steps S31-S33, and output the action command, and finally re-enter Step S4: The action instruction is compared with a preset action command. First, the present invention has an action instruction database to pre-record a preset action command, and each preset action command has a corresponding control command. For example, when the gesture command in the action command is a fist, the corresponding control command is closed. The electronic device; when there is an action part moving track command in the action command, the corresponding control command is to move the cursor according to the action part moving track command. In this step, the content of the action command outputted in step S3 is compared with the preset action command. If the action command content matches the preset action command, the control command corresponding to the preset action command is output to control the electronic device. .

In summary, the present invention provides an electronic device control system and method for controlling The system comprises an image pre-processing module, an image recognition module and an image comparison module; the control method comprises the steps of: S1: receiving a first motion image of a user; and S2: processing the first motion image to generate a second motion image; S3: defining an action part in the second action image, and identifying one action corresponding to the action part, and then outputting an action instruction formed by the action; and S4: setting the action instruction with a preset The motion instructions are compared. After receiving the first motion image, the image execution module performs step S2, and then the image recognition module performs steps S2 to S3, and finally the image comparison module executes step S4. Allowing a user to control the electronic device by limb motion and gesture action by the present invention.

Compared with the prior art, the present invention provides an electronic device control system and The method, because the human-machine interface is built in an embedded system of an ARM (Advanced RISC Machine) platform, so that as long as the electronic product with the receiving end is attached, the user can use the present invention to allow a user to only use the gesture action at a long distance. The way to replace the general remote control or touch function, and to achieve convenient, time-saving and low-cost effects.

The above description of the preferred embodiments of the present invention is intended to be illustrative of the invention and the scope of the invention. Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and it is to be understood that the invention may be modified and modified without departing from the spirit and scope of the invention. The scope is subject to the definition of the scope of the patent application attached.

1‧‧‧Electronic device control system

11‧‧‧Image capture device

12‧‧‧Image pre-processing module

13‧‧‧Command output module

14‧‧‧Image recognition module

16‧‧‧Image comparison module

Claims (12)

An electronic device control system for operating an electronic device by recognizing a first motion image of a user; comprising: an image pre-processing module, configured to pre-process the first motion image to generate a second motion An image recognition module electrically connected to the image pre-processing module for defining an action part of the second action image, and tracking and recognizing one of the actions corresponding to the action part, and then outputting the action Forming an action command; and an image comparison module electrically connected to the image recognition module, the image comparison module includes an action instruction database, wherein the action instruction database stores a preset action command, the image The comparison module is configured to compare whether the action instruction conforms to the preset action instruction, and if the comparison result is consistent, outputting a control instruction corresponding to one of the preset action instructions to control the electronic device. The electronic control device system of claim 1, wherein the action portion is any actionable part of the user's limb, such as a hand, a foot, an eye or a lip. The electronic control device system of claim 1, wherein the action comprises a palm movement track and a palm gesture, and the action command comprises a palm movement track command and a palm gesture command. The electronic control device system of claim 1, further comprising an image capture device electrically connected to the image pre-processing module for capturing the first motion image of the user and outputting To the image pre-processing module. The electronic control device system of claim 1, further comprising an instruction The output module is electrically connected to the image comparison module for transmitting the control command to the electronic device for controlling the electronic device. An electronic device control method includes the following steps: S1: receiving a first motion image of a user; S2: pre-processing the first motion image to generate a second motion image; and S3: in the second motion image Defining an action part, and identifying one action corresponding to the action part, and then outputting an action instruction formed by the action; and S4: comparing the action instruction with a preset action instruction; wherein, if the comparison If the result is met, the output controls the instruction corresponding to one of the preset action instructions. The electronic control device method of claim 6, wherein the step S2 further comprises the following substeps: S21: removing the static background in the first motion image and leaving at least one motion object image; S22: removing the At least one motion object image noise; S23: enhancing the at least one motion object image; and S24: numbering the at least one motion object image to form and output the second motion image including the at least one motion object image . The electronic control device method according to claim 7, wherein the method of performing step S21 includes a temporal difference subtraction method; the method of performing step S22 includes an erosion operation (Erosion); and the method of performing step S23 includes an expansion operation. (Dilation); The method of performing step S24 includes Connected Component Labeling. The electronic control device method of claim 7, wherein the step S3 further comprises Sub-step: S31: the number of the non-limb image in the image number of the at least one action object is removed, and the action part is further defined from the at least one action object image; S32: tracking the action part; and S33: identifying the action One of the parts forms an action and outputs an action command formed by the action. The electronic control device method of claim 9, wherein the method of performing step S31 comprises a similar ratio method and a Harris detection method; and the method of performing step S32 comprises a Histogram of Oriented Gradients (HOG). And an average displacement method (Mean Shift); the method of performing step S33 includes a temporal difference subtraction method (Temporal Difference) and a scanning projection method. The electronic control device method according to claim 6, wherein the action portion is any actionable part of the user's limb, such as a hand, a foot, an eye or a lip. The electronic control device method of claim 6, wherein the action comprises a palm movement track and a palm gesture, and the action command comprises a palm movement track command and a palm gesture command.